Jul 22 02:28:27.114798 ip-10-0-133-125 systemd[1]: Starting Kubernetes Kubelet... Jul 22 02:28:27.516032 ip-10-0-133-125 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:28:27.516032 ip-10-0-133-125 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 02:28:27.516032 ip-10-0-133-125 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:28:27.516032 ip-10-0-133-125 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 02:28:27.516032 ip-10-0-133-125 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 02:28:27.519611 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.519451 2549 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 02:28:27.524182 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524163 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524185 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524191 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524196 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524203 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524209 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524213 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524216 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524219 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524222 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524225 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524228 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524230 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524233 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.524230 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524236 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524239 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524241 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524244 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524246 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524252 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524256 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524259 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524262 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524264 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524267 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524270 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524274 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524278 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524314 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524331 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524335 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524339 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524343 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524347 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.524583 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524350 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524353 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524356 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524360 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524362 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524365 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524368 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524374 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524378 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524380 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524383 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524387 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524392 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524395 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524398 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524400 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524403 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524406 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524408 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524411 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.525053 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524414 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524417 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524420 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524422 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524426 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524429 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524432 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524436 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524439 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524442 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524445 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524447 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524450 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524452 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524455 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524458 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524461 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524464 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524466 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524469 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.525559 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524471 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524474 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524476 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524495 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524498 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524500 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524503 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524505 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524508 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524511 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524514 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524517 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524519 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524522 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524525 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524527 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524531 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524533 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524537 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.526031 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524544 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524549 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524552 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524554 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524558 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.524561 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525297 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525304 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525307 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525311 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525314 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525317 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525320 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525323 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525325 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525328 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525330 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525332 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525335 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.526478 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525339 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525344 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525347 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525349 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525352 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525354 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525357 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525359 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525361 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525364 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525366 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525368 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525371 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525375 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525377 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525380 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525383 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525386 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525388 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525390 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.527000 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525393 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525395 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525398 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525400 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525403 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525405 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525407 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525410 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525412 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525415 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525417 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525419 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525422 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525424 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525426 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525430 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525433 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525435 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525438 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.527503 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525440 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525443 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525445 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525448 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525450 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525453 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525456 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525458 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525461 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525464 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525467 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525469 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525472 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525474 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525477 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525500 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525503 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525505 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525508 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525511 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.527960 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525513 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525516 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525518 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525521 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525523 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525526 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525529 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525531 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525534 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525537 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525539 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525544 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525547 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525550 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525561 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525566 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525569 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525572 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525576 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.528433 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525580 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525583 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525585 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525589 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525592 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525595 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525598 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.525600 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526012 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526018 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526022 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526026 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526030 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526037 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526040 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526044 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526047 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526050 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526053 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.528906 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526056 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526058 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526061 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526064 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526067 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526069 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526072 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526075 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526077 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526081 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526083 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526086 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526089 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526091 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526094 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526097 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526099 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526102 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526105 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.529368 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526109 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526113 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526115 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526118 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526121 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526124 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526127 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526130 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526133 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526136 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526138 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526141 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526144 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526147 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526149 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526152 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526155 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526158 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526160 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526163 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.529851 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526165 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526168 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526171 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526173 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526176 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526179 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526181 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526184 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526186 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526189 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526192 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526194 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526197 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526199 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526202 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526204 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526207 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526209 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526212 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526215 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.530323 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526217 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526220 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526222 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526224 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526227 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526230 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526233 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526236 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526238 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526241 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526244 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526246 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526249 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526251 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526254 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526256 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526258 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526261 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526263 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526266 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.530823 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526268 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526271 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526273 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526276 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526279 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526281 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526284 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526286 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.526289 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526368 2549 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526387 2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526397 2549 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526403 2549 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526407 2549 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526410 2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526415 2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526420 2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526423 2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526426 2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526429 2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526433 2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526436 2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 02:28:27.531308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526439 2549 flags.go:64] FLAG: --cgroup-root="" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526442 2549 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526445 2549 flags.go:64] FLAG: --client-ca-file="" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526447 2549 flags.go:64] FLAG: --cloud-provider="external" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526451 2549 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526456 2549 flags.go:64] FLAG: --cluster-domain="" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526459 2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526462 2549 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526465 2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526469 2549 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526475 2549 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526491 2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526495 2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526498 2549 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526502 2549 flags.go:64] FLAG: --contention-profiling="false" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526505 2549 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526508 2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526512 2549 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526514 2549 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526519 2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526522 2549 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526525 2549 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526528 2549 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526531 2549 flags.go:64] FLAG: --enable-server="true" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526533 2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 02:28:27.531881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526538 2549 flags.go:64] FLAG: --event-burst="100" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526541 2549 flags.go:64] FLAG: --event-qps="50" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526544 2549 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526547 2549 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526550 2549 flags.go:64] FLAG: --eviction-hard="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526553 2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526556 2549 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526559 2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526562 2549 flags.go:64] FLAG: --eviction-soft="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526565 2549 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526570 2549 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526572 2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526575 2549 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526578 2549 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526581 2549 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526584 2549 flags.go:64] FLAG: --feature-gates="" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526588 2549 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526591 2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526594 2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526597 2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526600 2549 flags.go:64] FLAG: --healthz-port="10248" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526603 2549 flags.go:64] FLAG: --help="false" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526606 2549 flags.go:64] FLAG: --hostname-override="ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526609 2549 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 02:28:27.532476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526612 2549 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526615 2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526618 2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526621 2549 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526626 2549 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526629 2549 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526631 2549 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526634 2549 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526637 2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526641 2549 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526643 2549 flags.go:64] FLAG: --kube-reserved="" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526647 2549 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526650 2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526653 2549 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526656 2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526659 2549 flags.go:64] FLAG: --lock-file="" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526662 2549 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526665 2549 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526669 2549 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526674 2549 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526676 2549 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526680 2549 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526682 2549 flags.go:64] FLAG: --logging-format="text" Jul 22 02:28:27.533165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526685 2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526688 2549 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526691 2549 flags.go:64] FLAG: --manifest-url="" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526694 2549 flags.go:64] FLAG: --manifest-url-header="" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526699 2549 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526701 2549 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526706 2549 flags.go:64] FLAG: --max-pods="110" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526708 2549 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526712 2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526715 2549 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526718 2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526721 2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526724 2549 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526728 2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526736 2549 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526739 2549 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526742 2549 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526745 2549 flags.go:64] FLAG: --pod-cidr="" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526748 2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526753 2549 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526756 2549 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526760 2549 flags.go:64] FLAG: --pods-per-core="0" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526763 2549 flags.go:64] FLAG: --port="10250" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526765 2549 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 02:28:27.533794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526768 2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01249f40015535fbf" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526772 2549 flags.go:64] FLAG: --qos-reserved="" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526775 2549 flags.go:64] FLAG: --read-only-port="10255" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526778 2549 flags.go:64] FLAG: --register-node="true" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526782 2549 flags.go:64] FLAG: --register-with-taints="" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526785 2549 flags.go:64] FLAG: --registry-burst="10" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526788 2549 flags.go:64] FLAG: --registry-qps="5" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526791 2549 flags.go:64] FLAG: --reserved-cpus="" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526793 2549 flags.go:64] FLAG: --reserved-memory="" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526797 2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526800 2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526802 2549 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526805 2549 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526808 2549 flags.go:64] FLAG: --runonce="false" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526811 2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526814 2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526817 2549 flags.go:64] FLAG: --seccomp-default="false" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526820 2549 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526823 2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526826 2549 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526829 2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526833 2549 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526836 2549 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526839 2549 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526841 2549 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526844 2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 02:28:27.534592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526847 2549 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526850 2549 flags.go:64] FLAG: --system-cgroups="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526853 2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526858 2549 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526861 2549 flags.go:64] FLAG: --tls-cert-file="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526864 2549 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526868 2549 flags.go:64] FLAG: --tls-min-version="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526870 2549 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526873 2549 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526876 2549 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526880 2549 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526883 2549 flags.go:64] FLAG: --v="2" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526886 2549 flags.go:64] FLAG: --version="false" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526891 2549 flags.go:64] FLAG: --vmodule="" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526894 2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.526897 2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527021 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527025 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527028 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527031 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527034 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527036 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527039 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.535406 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527041 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527051 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527053 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527056 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527060 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527063 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527065 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527068 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527071 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527073 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527075 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527078 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527080 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527083 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527085 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527088 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527091 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527095 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527098 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527103 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.536153 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527106 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527109 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527112 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527115 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527117 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527119 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527122 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527124 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527127 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527129 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527132 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527135 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527137 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527140 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527150 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527154 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527159 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527162 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527164 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.536667 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527167 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527169 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527172 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527174 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527176 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527179 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527181 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527184 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527186 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527189 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527192 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527194 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527198 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527200 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527203 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527205 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527208 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527210 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527213 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527215 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.537121 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527218 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527220 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527222 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527225 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527227 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527230 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527233 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527235 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527242 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527246 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527248 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527251 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527254 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527256 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527258 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527261 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527263 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527266 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527269 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.537635 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527272 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527274 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527276 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527279 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527282 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527287 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527290 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527293 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527296 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527299 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527302 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527304 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527307 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.527309 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.528050 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:28:27.538088 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.537575 2549 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.537595 2549 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.537616 2549 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.537622 2549 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537684 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537691 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537694 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537698 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537701 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537704 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537706 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537709 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537713 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537716 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537719 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537721 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537724 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537727 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.538454 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537730 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537732 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537735 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537738 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537741 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537744 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537746 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537749 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537751 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537754 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537756 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537759 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537761 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537764 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537767 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537770 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537772 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537775 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537778 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537780 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.538934 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537783 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537786 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537788 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537792 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537795 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537798 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537801 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537804 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537806 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537808 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537811 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537814 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537816 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537818 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537821 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537824 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537826 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537828 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537831 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537833 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.539413 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537836 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537838 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537841 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537843 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537845 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537848 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537850 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537853 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537856 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537859 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537862 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537864 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537867 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537869 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537871 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537874 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537876 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537878 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537881 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537883 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.539976 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537886 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537888 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537891 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537893 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537895 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537899 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537901 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537904 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537906 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537908 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537911 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537913 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537915 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537918 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537920 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537923 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537925 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537928 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537930 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537933 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.540442 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537936 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537938 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537941 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537943 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.537945 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.537950 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538075 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538079 2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538082 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538085 2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538088 2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538091 2549 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538094 2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538097 2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 02:28:27.540972 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538099 2549 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538102 2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538105 2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538107 2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538110 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538113 2549 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538116 2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538120 2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538123 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538125 2549 feature_gate.go:328] unrecognized feature gate: Example Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538128 2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538130 2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538132 2549 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538135 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538137 2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538140 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538142 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538144 2549 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538147 2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538150 2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 02:28:27.541351 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538153 2549 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538156 2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538158 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538160 2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538163 2549 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538165 2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538167 2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538170 2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538172 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538176 2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538179 2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538182 2549 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538184 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538187 2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538189 2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538191 2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538194 2549 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538197 2549 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538200 2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 02:28:27.541845 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538202 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538205 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538207 2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538210 2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538213 2549 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538215 2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538217 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538220 2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538222 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538225 2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538227 2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538230 2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538232 2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538235 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538237 2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538240 2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538242 2549 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538245 2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538247 2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538250 2549 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 02:28:27.542291 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538252 2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538255 2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538257 2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538259 2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538262 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538264 2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538267 2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538269 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538272 2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538274 2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538276 2549 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538279 2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538283 2549 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538286 2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538289 2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538292 2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538295 2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538297 2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538300 2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538302 2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 02:28:27.542879 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538304 2549 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538307 2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538309 2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538312 2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538314 2549 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538317 2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538319 2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538322 2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538324 2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538327 2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538329 2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:27.538331 2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.538336 2549 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.538461 2549 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.540737 2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 02:28:27.543381 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.542291 2549 server.go:1019] "Starting client certificate rotation" Jul 22 02:28:27.543770 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.542410 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:28:27.543770 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.542461 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 02:28:27.544872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.544855 2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:28:27.546163 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.546142 2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 02:28:27.556524 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.556507 2549 log.go:25] "Validated CRI v1 runtime API" Jul 22 02:28:27.558726 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.558711 2549 log.go:25] "Validated CRI v1 image API" Jul 22 02:28:27.560419 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.560400 2549 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 02:28:27.567872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.567847 2549 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7def3029-eaf3-411e-aa52-9836ee36852f:/dev/nvme0n1p3 8050bb38-9f0a-4c46-a8fd-7b5fccaf4f7a:/dev/nvme0n1p4] Jul 22 02:28:27.567979 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.567868 2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 02:28:27.573667 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.573506 2549 manager.go:216] Machine: {Timestamp:2026-07-22 02:28:27.571824916 +0000 UTC m=+0.338767053 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102286 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec227e964a2b1bea61c633e2fb8c2f6f SystemUUID:ec227e96-4a2b-1bea-61c6-33e2fb8c2f6f BootID:2876cf80-cb25-433b-9916-c5823b47fabb Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1d:39:41:15:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1d:39:41:15:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:ae:d1:5f:a6:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 02:28:27.573667 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.573652 2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 02:28:27.573813 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.573737 2549 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 02:28:27.576882 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.576852 2549 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 02:28:27.577028 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.576886 2549 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 02:28:27.578291 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.578278 2549 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 02:28:27.578333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.578293 2549 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 02:28:27.578333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.578301 2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:28:27.578985 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.578974 2549 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 02:28:27.580188 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.580174 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:28:27.580509 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.580498 2549 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 02:28:27.583814 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583801 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:28:27.583935 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583925 2549 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 02:28:27.583966 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583957 2549 kubelet.go:486] "Attempting to sync node with API server" Jul 22 02:28:27.584004 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583968 2549 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 02:28:27.584004 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583981 2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 02:28:27.584004 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583991 2549 kubelet.go:398] "Adding apiserver pod source" Jul 22 02:28:27.584004 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.583999 2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 02:28:27.584426 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.584414 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:28:27.584465 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.584433 2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 02:28:27.586762 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.586742 2549 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 02:28:27.588752 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.588738 2549 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 02:28:27.588796 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.588771 2549 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 02:28:27.590398 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590387 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 02:28:27.590434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590402 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 02:28:27.590434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590408 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 02:28:27.590434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590414 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 02:28:27.590434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590423 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 02:28:27.590434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590430 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590439 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590445 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590455 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590464 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590499 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 02:28:27.590585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.590508 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 02:28:27.591271 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.591259 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 02:28:27.591300 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.591274 2549 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 02:28:27.594175 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.594152 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 02:28:27.594255 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.594166 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 02:28:27.594255 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.594171 2549 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 02:28:27.594863 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.594850 2549 server.go:1268] "Started kubelet" Jul 22 02:28:27.594984 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.594951 2549 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 02:28:27.595098 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.594921 2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 02:28:27.595098 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.595056 2549 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 02:28:27.595741 ip-10-0-133-125 systemd[1]: Started Kubernetes Kubelet. Jul 22 02:28:27.597019 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.597001 2549 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 02:28:27.597099 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.597083 2549 server.go:310] "Adding debug handlers to kubelet server" Jul 22 02:28:27.601319 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.601295 2549 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 02:28:27.604777 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.602079 2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-125.ec2.internal.18c47ba68c566b08 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-125.ec2.internal,UID:ip-10-0-133-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-125.ec2.internal,},FirstTimestamp:2026-07-22 02:28:27.59482036 +0000 UTC m=+0.361762496,LastTimestamp:2026-07-22 02:28:27.59482036 +0000 UTC m=+0.361762496,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-125.ec2.internal,}" Jul 22 02:28:27.605636 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.605618 2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 02:28:27.605636 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.605627 2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 02:28:27.605859 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.605835 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:27.606397 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606380 2549 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 02:28:27.606510 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606386 2549 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 02:28:27.606510 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606420 2549 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 02:28:27.606614 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606515 2549 factory.go:55] Registering systemd factory Jul 22 02:28:27.606614 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606529 2549 factory.go:223] Registration of the systemd container factory successfully Jul 22 02:28:27.606614 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606577 2549 reconstruct.go:97] "Volume reconstruction finished" Jul 22 02:28:27.606614 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606589 2549 reconciler.go:29] "Reconciler: start to sync state" Jul 22 02:28:27.606977 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606959 2549 factory.go:153] Registering CRI-O factory Jul 22 02:28:27.606977 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.606978 2549 factory.go:223] Registration of the crio container factory successfully Jul 22 02:28:27.607112 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.607039 2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 02:28:27.607112 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.607063 2549 factory.go:103] Registering Raw factory Jul 22 02:28:27.607112 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.607078 2549 manager.go:1196] Started watching for new ooms in manager Jul 22 02:28:27.607606 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.607594 2549 manager.go:349] Starting recovery of all containers Jul 22 02:28:27.608043 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.608021 2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 02:28:27.608114 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.608078 2549 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 02:28:27.619277 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.619142 2549 manager.go:354] Recovery completed Jul 22 02:28:27.621034 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.621003 2549 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 22 02:28:27.621754 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.621684 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4kfl" Jul 22 02:28:27.624203 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.624191 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.626082 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626068 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.626146 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626098 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.626146 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626109 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.626727 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626713 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.626802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626716 2549 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 02:28:27.626802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626748 2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 02:28:27.626802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.626763 2549 state_mem.go:36] "Initialized new in-memory state store" Jul 22 02:28:27.627318 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.627303 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.627372 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.627332 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.627372 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.627348 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.628219 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.628208 2549 policy_none.go:49] "None policy: Start" Jul 22 02:28:27.628274 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.628223 2549 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 02:28:27.628274 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.628233 2549 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 02:28:27.628635 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.628621 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4kfl" Jul 22 02:28:27.630198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.630184 2549 policy_none.go:47] "Start" Jul 22 02:28:27.666115 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666089 2549 manager.go:337] "Starting Device Plugin manager" Jul 22 02:28:27.666239 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.666139 2549 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 02:28:27.666239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666152 2549 server.go:87] "Starting device plugin registration server" Jul 22 02:28:27.666451 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666436 2549 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 02:28:27.666528 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666455 2549 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 02:28:27.666588 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666570 2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 02:28:27.666670 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666644 2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 02:28:27.666670 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.666659 2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 02:28:27.667183 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.667162 2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 02:28:27.667245 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.667213 2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:27.761098 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.761051 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 02:28:27.762418 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.762404 2549 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 02:28:27.762512 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.762449 2549 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 02:28:27.762512 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.762476 2549 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 02:28:27.762604 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.762539 2549 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 02:28:27.765187 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.765169 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:28:27.767373 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.767336 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.768086 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.768073 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.768137 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.768104 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.768137 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.768113 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.768137 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.768135 2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.776445 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.776422 2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.776549 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.776449 2549 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-125.ec2.internal\": node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:27.807039 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.807014 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:27.863069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.863032 2549 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal"] Jul 22 02:28:27.863195 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.863119 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.864820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.864803 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.864920 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.864841 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.864920 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.864854 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.866296 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.866280 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.866451 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.866438 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.866518 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.866498 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.867084 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867070 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.867148 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867096 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.867148 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867106 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.867837 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867823 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.867934 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867855 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.867934 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.867867 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.869077 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.869064 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.869125 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.869087 2549 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 02:28:27.869737 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.869716 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientMemory" Jul 22 02:28:27.869824 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.869746 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 02:28:27.869824 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:27.869760 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeHasSufficientPID" Jul 22 02:28:27.895174 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.895148 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.899924 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.899895 2549 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-125.ec2.internal\" not found" node="ip-10-0-133-125.ec2.internal" Jul 22 02:28:27.908021 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:27.908004 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.007140 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.007108 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.007309 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.007146 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.007309 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.007171 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5786553615a694db0fe10013f4d054ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"5786553615a694db0fe10013f4d054ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.008132 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.008114 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.108014 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.107914 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108014 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.107966 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108014 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.107990 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108014 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.108000 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5786553615a694db0fe10013f4d054ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"5786553615a694db0fe10013f4d054ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108223 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.108026 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5786553615a694db0fe10013f4d054ba-config\") pod \"kube-apiserver-proxy-ip-10-0-133-125.ec2.internal\" (UID: \"5786553615a694db0fe10013f4d054ba\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108223 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.108069 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5c509b29f0f55da12efeeb271236c0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal\" (UID: \"d5c509b29f0f55da12efeeb271236c0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.108972 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.108954 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.199218 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.199177 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.202787 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.202767 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:28.209447 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.209432 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.309898 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.309860 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.410422 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.410339 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.510805 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:28.510767 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:28.541963 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.541931 2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 02:28:28.542418 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:28.542080 2549 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 02:28:30.221029 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.221001 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.243751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.243711 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 02:23:29 +0000 UTC" deadline="2028-01-24 09:17:42.478319527 +0000 UTC" Jul 22 02:28:30.243751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.243740 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13230h49m12.234582101s" Jul 22 02:28:30.287211 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.287150 2549 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:28:30.322103 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.322071 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.422487 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.422451 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.435264 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:30.435224 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c509b29f0f55da12efeeb271236c0a.slice/crio-a2b62078dd4a9a956119bb54a56ab0e56d380b558f9b42ef34887ef0f502315f WatchSource:0}: Error finding container a2b62078dd4a9a956119bb54a56ab0e56d380b558f9b42ef34887ef0f502315f: Status 404 returned error can't find the container with id a2b62078dd4a9a956119bb54a56ab0e56d380b558f9b42ef34887ef0f502315f Jul 22 02:28:30.435522 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:30.435502 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5786553615a694db0fe10013f4d054ba.slice/crio-6eee42b524aeff9b968ce5bbddcb705e5a1c0dd75574139369d1686598177290 WatchSource:0}: Error finding container 6eee42b524aeff9b968ce5bbddcb705e5a1c0dd75574139369d1686598177290: Status 404 returned error can't find the container with id 6eee42b524aeff9b968ce5bbddcb705e5a1c0dd75574139369d1686598177290 Jul 22 02:28:30.440231 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.440214 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:28:30.522765 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.522708 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.536054 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.536033 2549 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:28:30.623223 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.623125 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.723890 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:30.723861 2549 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-125.ec2.internal\" not found" Jul 22 02:28:30.752606 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.752578 2549 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:28:30.816618 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.816575 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" Jul 22 02:28:30.834898 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.834876 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:28:30.835985 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.835972 2549 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" Jul 22 02:28:30.842611 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:30.842592 2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 02:28:31.174547 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.174346 2549 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 02:28:31.194935 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.194905 2549 apiserver.go:52] "Watching apiserver" Jul 22 02:28:31.204432 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.204383 2549 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 02:28:31.206670 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.206631 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-dctdr","openshift-ovn-kubernetes/ovnkube-node-2xxp6","kube-system/global-pull-secret-syncer-q6k5b","kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth","openshift-cluster-node-tuning-operator/tuned-klzgp","openshift-dns/node-resolver-qbfxt","openshift-image-registry/node-ca-j8qnh","openshift-multus/multus-additional-cni-plugins-q4cnn","openshift-multus/network-metrics-daemon-lq5wc","kube-system/konnectivity-agent-r45jl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal","openshift-multus/multus-pp4ls","openshift-network-diagnostics/network-check-target-wcgkq"] Jul 22 02:28:31.208653 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.208630 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.208754 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.208720 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:31.210558 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.210537 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.214999 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.214978 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.215080 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.215041 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:31.215123 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215101 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:31.215155 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.215129 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:31.215500 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215397 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 02:28:31.215500 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215447 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 02:28:31.215601 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215525 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.215601 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215531 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k6wz2\"" Jul 22 02:28:31.215601 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215453 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.215720 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.215710 2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 02:28:31.216210 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.216191 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.218097 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.218079 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.220113 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.219818 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.221282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.221250 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.223344 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.223326 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.223466 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.223335 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 02:28:31.224211 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.224193 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.225198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.225178 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.225198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.225196 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fbh22\"" Jul 22 02:28:31.225350 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.225327 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.226114 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.226087 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-828xl\"" Jul 22 02:28:31.226358 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.226337 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.226471 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.226455 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.226793 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.226774 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hs8ng\"" Jul 22 02:28:31.226966 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.226946 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.227032 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227018 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 02:28:31.227318 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227300 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 02:28:31.227678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227654 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-device-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.227752 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227698 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmhb\" (UniqueName: \"kubernetes.io/projected/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kube-api-access-dxmhb\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.227752 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227734 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-kubernetes\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.227847 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227774 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ksbpp\"" Jul 22 02:28:31.227847 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227757 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-conf\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.227954 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227873 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ngr\" (UniqueName: \"kubernetes.io/projected/d191d66b-70ec-41d9-a66c-dc564775ae3f-kube-api-access-v2ngr\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.227954 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.227900 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-hostroot\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228062 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228007 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdmgx\" (UniqueName: \"kubernetes.io/projected/bdeb088c-7753-438d-a41d-6bf69688c907-kube-api-access-kdmgx\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228062 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228037 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-socket-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.228168 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228064 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-etc-selinux\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.228224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228168 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-sys\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.228224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228196 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-tmp\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.228324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228224 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-k8s-cni-cncf-io\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228250 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-kubelet\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228275 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-conf-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228304 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-dbus\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228332 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysconfig\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228373 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f9z4\" (UniqueName: \"kubernetes.io/projected/c434dc6a-0628-43fe-9915-263b2bbd3a64-kube-api-access-9f9z4\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228384 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228429 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-netns\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228473 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-etc-kubernetes\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228520 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.228833 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228750 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.228833 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228817 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-var-lib-kubelet\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.228931 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228848 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-system-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.228931 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228871 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-os-release\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.229025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228942 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-multus-certs\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.229025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.228996 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-kubelet-config\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.229134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229030 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-registration-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.229134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229084 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-modprobe-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.229134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229118 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-lib-modules\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.229266 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229167 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-tuned\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.229359 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229331 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-multus\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.229478 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229399 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-multus-daemon-config\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.229478 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229465 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.229587 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229497 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:31.229587 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229579 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-host\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.229715 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229699 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-bin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.229763 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229744 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-sys-fs\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.229802 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-systemd\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230156 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-run\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230177 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230203 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230219 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-cni-binary-copy\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230232 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230255 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-cnibin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.230333 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230270 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-socket-dir-parent\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.230732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230682 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.230732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.230720 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.232438 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232181 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 02:28:31.232438 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232189 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 02:28:31.232438 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232338 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.232438 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232350 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.232595 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232453 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jw58b\"" Jul 22 02:28:31.232595 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232554 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 02:28:31.232820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232714 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47dqn\"" Jul 22 02:28:31.232820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232781 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.232820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232819 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 02:28:31.233098 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.232981 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 02:28:31.234540 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.234522 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 02:28:31.235134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.235042 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 02:28:31.235261 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.235243 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d88sn\"" Jul 22 02:28:31.235520 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.235500 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 02:28:31.235866 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.235850 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 02:28:31.236638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.236620 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 02:28:31.238159 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.238141 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpbfz\"" Jul 22 02:28:31.250169 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.250145 2549 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 02:28:31.279046 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.279023 2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2xqs5" Jul 22 02:28:31.291599 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.291582 2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2xqs5" Jul 22 02:28:31.317790 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.317749 2549 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 02:28:31.331017 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.330993 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-kubelet\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331031 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95b795f1-87af-4837-b0c7-6c2d701f34e3-iptables-alerter-script\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.331134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331056 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-log-socket\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.331134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331095 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-netns\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331123 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-kubelet\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331130 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-etc-kubernetes\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331164 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-modprobe-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331178 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-netns\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331188 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-lib-modules\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331167 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-etc-kubernetes\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331225 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-tuned\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331248 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331252 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-modprobe-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331270 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-multus-daemon-config\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331299 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-env-overrides\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331300 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-lib-modules\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331321 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-os-release\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331348 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-bin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331374 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-run\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331395 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-cnibin\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331434 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-cni-binary-copy\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331514 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-bin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331563 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abf512bb-898b-4547-8286-6219249d80f9-agent-certs\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331570 2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331598 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-cnibin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331626 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-device-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331652 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-conf\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331678 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvr9\" (UniqueName: \"kubernetes.io/projected/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-kube-api-access-5qvr9\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331702 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-var-lib-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331731 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-node-log\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331783 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-script-lib\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331800 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-multus-daemon-config\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331809 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.331872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331839 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdmgx\" (UniqueName: \"kubernetes.io/projected/bdeb088c-7753-438d-a41d-6bf69688c907-kube-api-access-kdmgx\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331865 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-socket-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331888 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-etc-selinux\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331903 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-cnibin\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331910 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-tmp\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331933 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-host\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331936 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-device-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331955 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-kubelet\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.331978 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-systemd-units\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332012 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bdeb088c-7753-438d-a41d-6bf69688c907-cni-binary-copy\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332026 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-conf\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332003 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332061 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-k8s-cni-cncf-io\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332072 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-run\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332102 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-socket-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332104 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-conf-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332128 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-dbus\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.332680 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332229 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-etc-selinux\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332344 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysconfig\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332377 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f9z4\" (UniqueName: \"kubernetes.io/projected/c434dc6a-0628-43fe-9915-263b2bbd3a64-kube-api-access-9f9z4\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332405 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-config\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332448 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332471 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-var-lib-kubelet\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332492 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvbz\" (UniqueName: \"kubernetes.io/projected/95b795f1-87af-4837-b0c7-6c2d701f34e3-kube-api-access-rkvbz\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332543 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkx4w\" (UniqueName: \"kubernetes.io/projected/f8641f99-d34b-4f4e-8af0-22b380077196-kube-api-access-bkx4w\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332567 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f601a1-dfc7-45a1-9028-0b514b932161-tmp-dir\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332614 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysconfig\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332662 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-k8s-cni-cncf-io\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332601 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-system-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332716 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-conf-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332790 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-os-release\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332830 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332876 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-multus-certs\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332904 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-kubelet-config\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.333555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332924 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-run-multus-certs\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332929 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-registration-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332957 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95b795f1-87af-4837-b0c7-6c2d701f34e3-host-slash\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332980 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-serviceca\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.332988 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-os-release\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333031 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-systemd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333060 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-multus\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333072 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-system-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333089 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333120 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333146 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-host\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333164 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-dbus\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333199 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-bin\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333243 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.333265 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333271 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.333340 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:31.833317255 +0000 UTC m=+2.990329538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:31.334282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333433 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a35637e-f5c1-452f-9b43-46f19f25314f-kubelet-config\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333458 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-sys-fs\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333488 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-host\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333491 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-host-var-lib-cni-multus\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333490 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-systemd\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333531 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-registration-dir\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333580 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-systemd\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333611 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333631 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-sys-fs\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333655 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.333709 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333741 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-netd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333767 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmw4\" (UniqueName: \"kubernetes.io/projected/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-kube-api-access-fxmw4\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.333800 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:31.833780937 +0000 UTC m=+2.990793216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333827 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-system-cni-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333871 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333919 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333937 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abf512bb-898b-4547-8286-6219249d80f9-konnectivity-ca\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333961 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-socket-dir-parent\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333977 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmhb\" (UniqueName: \"kubernetes.io/projected/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kube-api-access-dxmhb\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.333999 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-cni-dir\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334003 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-multus-socket-dir-parent\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334019 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-kubernetes\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334036 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ngr\" (UniqueName: \"kubernetes.io/projected/d191d66b-70ec-41d9-a66c-dc564775ae3f-kube-api-access-v2ngr\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334079 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-sysctl-d\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334135 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-netns\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334142 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-var-lib-kubelet\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334153 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-ovn\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334203 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334209 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-kubernetes\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334233 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxhc\" (UniqueName: \"kubernetes.io/projected/50f601a1-dfc7-45a1-9028-0b514b932161-kube-api-access-fcxhc\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334253 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-slash\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334268 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-hostroot\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334283 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-sys\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.335690 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334297 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-etc-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334325 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334331 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bdeb088c-7753-438d-a41d-6bf69688c907-hostroot\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334351 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f601a1-dfc7-45a1-9028-0b514b932161-hosts-file\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.334366 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c434dc6a-0628-43fe-9915-263b2bbd3a64-sys\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.335137 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-etc-tuned\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.336370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.335567 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c434dc6a-0628-43fe-9915-263b2bbd3a64-tmp\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.340326 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.339958 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:31.340326 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.340030 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:31.340326 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.340044 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:31.340326 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.340104 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:28:31.840085416 +0000 UTC m=+2.997097723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:31.346346 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.346307 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmhb\" (UniqueName: \"kubernetes.io/projected/7ac5ebfc-935d-47f0-9d0d-ba27668f791f-kube-api-access-dxmhb\") pod \"aws-ebs-csi-driver-node-swhth\" (UID: \"7ac5ebfc-935d-47f0-9d0d-ba27668f791f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.347284 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.347256 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ngr\" (UniqueName: \"kubernetes.io/projected/d191d66b-70ec-41d9-a66c-dc564775ae3f-kube-api-access-v2ngr\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.347716 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.347693 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdmgx\" (UniqueName: \"kubernetes.io/projected/bdeb088c-7753-438d-a41d-6bf69688c907-kube-api-access-kdmgx\") pod \"multus-pp4ls\" (UID: \"bdeb088c-7753-438d-a41d-6bf69688c907\") " pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.349251 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.349227 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f9z4\" (UniqueName: \"kubernetes.io/projected/c434dc6a-0628-43fe-9915-263b2bbd3a64-kube-api-access-9f9z4\") pod \"tuned-klzgp\" (UID: \"c434dc6a-0628-43fe-9915-263b2bbd3a64\") " pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.376573 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.376523 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"d5c509b29f0f55da12efeeb271236c0a","Type":"ContainerStarted","Data":"a2b62078dd4a9a956119bb54a56ab0e56d380b558f9b42ef34887ef0f502315f"} Jul 22 02:28:31.377540 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.377516 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"5786553615a694db0fe10013f4d054ba","Type":"ContainerStarted","Data":"6eee42b524aeff9b968ce5bbddcb705e5a1c0dd75574139369d1686598177290"} Jul 22 02:28:31.434684 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434590 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvr9\" (UniqueName: \"kubernetes.io/projected/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-kube-api-access-5qvr9\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.434684 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434633 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-var-lib-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434684 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434655 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-node-log\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434684 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434678 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-script-lib\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434728 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-node-log\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434736 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-var-lib-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434759 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434789 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-host\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434811 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-kubelet\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434833 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-systemd-units\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434859 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434887 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-config\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434903 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-host\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434917 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvbz\" (UniqueName: \"kubernetes.io/projected/95b795f1-87af-4837-b0c7-6c2d701f34e3-kube-api-access-rkvbz\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434938 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkx4w\" (UniqueName: \"kubernetes.io/projected/f8641f99-d34b-4f4e-8af0-22b380077196-kube-api-access-bkx4w\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434945 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-kubelet\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434961 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f601a1-dfc7-45a1-9028-0b514b932161-tmp-dir\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.434982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.434962 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435066 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435094 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-systemd-units\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435143 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95b795f1-87af-4837-b0c7-6c2d701f34e3-host-slash\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435172 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-serviceca\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435200 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-systemd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435217 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/95b795f1-87af-4837-b0c7-6c2d701f34e3-host-slash\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435249 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-bin\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435272 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-systemd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435273 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435306 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50f601a1-dfc7-45a1-9028-0b514b932161-tmp-dir\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435312 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435354 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435352 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-bin\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435378 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-script-lib\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435404 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435441 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-netd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.435628 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435471 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmw4\" (UniqueName: \"kubernetes.io/projected/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-kube-api-access-fxmw4\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435499 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-system-cni-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435527 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abf512bb-898b-4547-8286-6219249d80f9-konnectivity-ca\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435532 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-cni-netd\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435547 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovnkube-config\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435556 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-netns\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435575 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-system-cni-dir\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435581 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-ovn\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435611 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435635 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxhc\" (UniqueName: \"kubernetes.io/projected/50f601a1-dfc7-45a1-9028-0b514b932161-kube-api-access-fcxhc\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435650 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-slash\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435666 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-etc-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435681 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435679 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-serviceca\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435708 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f601a1-dfc7-45a1-9028-0b514b932161-hosts-file\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435739 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-run-netns\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435748 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95b795f1-87af-4837-b0c7-6c2d701f34e3-iptables-alerter-script\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.436376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435774 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-log-socket\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435788 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-etc-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435802 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435810 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50f601a1-dfc7-45a1-9028-0b514b932161-hosts-file\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435823 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-env-overrides\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435844 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-os-release\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435859 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-openvswitch\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435871 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-cnibin\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435924 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-cnibin\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435928 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-host-slash\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.435959 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abf512bb-898b-4547-8286-6219249d80f9-agent-certs\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436118 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abf512bb-898b-4547-8286-6219249d80f9-konnectivity-ca\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436169 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436175 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-run-ovn\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436233 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8641f99-d34b-4f4e-8af0-22b380077196-os-release\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436243 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-log-socket\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436392 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436504 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-env-overrides\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.437681 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.436528 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8641f99-d34b-4f4e-8af0-22b380077196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.437681 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.437334 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/95b795f1-87af-4837-b0c7-6c2d701f34e3-iptables-alerter-script\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.437939 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.437920 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-ovn-node-metrics-cert\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.438511 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.438489 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abf512bb-898b-4547-8286-6219249d80f9-agent-certs\") pod \"konnectivity-agent-r45jl\" (UID: \"abf512bb-898b-4547-8286-6219249d80f9\") " pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.445523 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.445498 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvbz\" (UniqueName: \"kubernetes.io/projected/95b795f1-87af-4837-b0c7-6c2d701f34e3-kube-api-access-rkvbz\") pod \"iptables-alerter-dctdr\" (UID: \"95b795f1-87af-4837-b0c7-6c2d701f34e3\") " pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.446462 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.446443 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxhc\" (UniqueName: \"kubernetes.io/projected/50f601a1-dfc7-45a1-9028-0b514b932161-kube-api-access-fcxhc\") pod \"node-resolver-qbfxt\" (UID: \"50f601a1-dfc7-45a1-9028-0b514b932161\") " pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.446858 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.446836 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvr9\" (UniqueName: \"kubernetes.io/projected/b0a71a19-21a0-4493-b9ae-c8b2c8e9845c-kube-api-access-5qvr9\") pod \"node-ca-j8qnh\" (UID: \"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c\") " pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.447371 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.447350 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmw4\" (UniqueName: \"kubernetes.io/projected/e6f93fc3-b381-431c-a247-d2c9c4c31f2c-kube-api-access-fxmw4\") pod \"ovnkube-node-2xxp6\" (UID: \"e6f93fc3-b381-431c-a247-d2c9c4c31f2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.449082 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.449064 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkx4w\" (UniqueName: \"kubernetes.io/projected/f8641f99-d34b-4f4e-8af0-22b380077196-kube-api-access-bkx4w\") pod \"multus-additional-cni-plugins-q4cnn\" (UID: \"f8641f99-d34b-4f4e-8af0-22b380077196\") " pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.526659 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.526626 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pp4ls" Jul 22 02:28:31.536996 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.536976 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" Jul 22 02:28:31.546375 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.546350 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-klzgp" Jul 22 02:28:31.552303 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.551991 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" Jul 22 02:28:31.560505 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.560482 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qbfxt" Jul 22 02:28:31.567551 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.567531 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:31.576225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.576190 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dctdr" Jul 22 02:28:31.583643 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.583495 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:31.590638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.590618 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j8qnh" Jul 22 02:28:31.838091 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.838009 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:31.838091 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.838060 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:31.838288 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.838191 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:31.838288 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.838254 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:32.838240066 +0000 UTC m=+3.995252330 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:31.838379 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.838191 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:31.838379 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.838333 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:32.838321254 +0000 UTC m=+3.995333518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:31.939295 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:31.939245 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:31.939503 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.939440 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:31.939503 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.939466 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:31.939503 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.939476 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:31.939634 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:31.939539 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:28:32.939525514 +0000 UTC m=+4.096537801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:32.294119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.294073 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:23:31 +0000 UTC" deadline="2028-02-01 04:33:42.727199537 +0000 UTC" Jul 22 02:28:32.294119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.294119 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13418h5m10.433084752s" Jul 22 02:28:32.359168 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.359140 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f93fc3_b381_431c_a247_d2c9c4c31f2c.slice/crio-c5304b79c8beb6324d427d3cccd90f3e6f032b3944f5f1309a34dccad1ce9327 WatchSource:0}: Error finding container c5304b79c8beb6324d427d3cccd90f3e6f032b3944f5f1309a34dccad1ce9327: Status 404 returned error can't find the container with id c5304b79c8beb6324d427d3cccd90f3e6f032b3944f5f1309a34dccad1ce9327 Jul 22 02:28:32.360229 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.360204 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf512bb_898b_4547_8286_6219249d80f9.slice/crio-b92e1e44fb97840343ec061b2b1487bf33be99820b46ad2796d102f3e7fa3089 WatchSource:0}: Error finding container b92e1e44fb97840343ec061b2b1487bf33be99820b46ad2796d102f3e7fa3089: Status 404 returned error can't find the container with id b92e1e44fb97840343ec061b2b1487bf33be99820b46ad2796d102f3e7fa3089 Jul 22 02:28:32.362092 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.362003 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeb088c_7753_438d_a41d_6bf69688c907.slice/crio-0a569f115ea2aa46c3a61791c250569851369233d3b9d431293a45eb4e80af1b WatchSource:0}: Error finding container 0a569f115ea2aa46c3a61791c250569851369233d3b9d431293a45eb4e80af1b: Status 404 returned error can't find the container with id 0a569f115ea2aa46c3a61791c250569851369233d3b9d431293a45eb4e80af1b Jul 22 02:28:32.365595 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.365527 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac5ebfc_935d_47f0_9d0d_ba27668f791f.slice/crio-6ba73fb9f2c73291558540e25b99883c50cb09948f67712e23b18d5a81aedecd WatchSource:0}: Error finding container 6ba73fb9f2c73291558540e25b99883c50cb09948f67712e23b18d5a81aedecd: Status 404 returned error can't find the container with id 6ba73fb9f2c73291558540e25b99883c50cb09948f67712e23b18d5a81aedecd Jul 22 02:28:32.366910 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.366619 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f601a1_dfc7_45a1_9028_0b514b932161.slice/crio-aaaaa80739c78aa36334918c396fe88f8718146a565ffbefe90f68c8e3604fd8 WatchSource:0}: Error finding container aaaaa80739c78aa36334918c396fe88f8718146a565ffbefe90f68c8e3604fd8: Status 404 returned error can't find the container with id aaaaa80739c78aa36334918c396fe88f8718146a565ffbefe90f68c8e3604fd8 Jul 22 02:28:32.367988 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.367907 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8641f99_d34b_4f4e_8af0_22b380077196.slice/crio-58e650b02dbc5b9d7aba57d63100a0809f76afe785541381386f1e4cbe8d4ecb WatchSource:0}: Error finding container 58e650b02dbc5b9d7aba57d63100a0809f76afe785541381386f1e4cbe8d4ecb: Status 404 returned error can't find the container with id 58e650b02dbc5b9d7aba57d63100a0809f76afe785541381386f1e4cbe8d4ecb Jul 22 02:28:32.368957 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.368881 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b795f1_87af_4837_b0c7_6c2d701f34e3.slice/crio-b8fe90112607bcda6b7e7a47a68b460da870ab2ea04597e2da828648dffc7966 WatchSource:0}: Error finding container b8fe90112607bcda6b7e7a47a68b460da870ab2ea04597e2da828648dffc7966: Status 404 returned error can't find the container with id b8fe90112607bcda6b7e7a47a68b460da870ab2ea04597e2da828648dffc7966 Jul 22 02:28:32.369240 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.369221 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc434dc6a_0628_43fe_9915_263b2bbd3a64.slice/crio-be78a005883b23148c9836bcab168f66128fbab8fc7aa8372cbf3c367c5be1a0 WatchSource:0}: Error finding container be78a005883b23148c9836bcab168f66128fbab8fc7aa8372cbf3c367c5be1a0: Status 404 returned error can't find the container with id be78a005883b23148c9836bcab168f66128fbab8fc7aa8372cbf3c367c5be1a0 Jul 22 02:28:32.371644 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:28:32.371620 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a71a19_21a0_4493_b9ae_c8b2c8e9845c.slice/crio-3f711f121f91ab06130ddb603f916a171396aca72456537ce817940bd1556950 WatchSource:0}: Error finding container 3f711f121f91ab06130ddb603f916a171396aca72456537ce817940bd1556950: Status 404 returned error can't find the container with id 3f711f121f91ab06130ddb603f916a171396aca72456537ce817940bd1556950 Jul 22 02:28:32.373004 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.372858 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:32.373004 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.372964 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:32.379899 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.379876 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klzgp" event={"ID":"c434dc6a-0628-43fe-9915-263b2bbd3a64","Type":"ContainerStarted","Data":"be78a005883b23148c9836bcab168f66128fbab8fc7aa8372cbf3c367c5be1a0"} Jul 22 02:28:32.381301 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.381278 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qbfxt" event={"ID":"50f601a1-dfc7-45a1-9028-0b514b932161","Type":"ContainerStarted","Data":"aaaaa80739c78aa36334918c396fe88f8718146a565ffbefe90f68c8e3604fd8"} Jul 22 02:28:32.382372 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.382352 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pp4ls" event={"ID":"bdeb088c-7753-438d-a41d-6bf69688c907","Type":"ContainerStarted","Data":"0a569f115ea2aa46c3a61791c250569851369233d3b9d431293a45eb4e80af1b"} Jul 22 02:28:32.383544 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.383519 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8qnh" event={"ID":"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c","Type":"ContainerStarted","Data":"3f711f121f91ab06130ddb603f916a171396aca72456537ce817940bd1556950"} Jul 22 02:28:32.384617 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.384587 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerStarted","Data":"58e650b02dbc5b9d7aba57d63100a0809f76afe785541381386f1e4cbe8d4ecb"} Jul 22 02:28:32.385544 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.385509 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" event={"ID":"7ac5ebfc-935d-47f0-9d0d-ba27668f791f","Type":"ContainerStarted","Data":"6ba73fb9f2c73291558540e25b99883c50cb09948f67712e23b18d5a81aedecd"} Jul 22 02:28:32.386513 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.386493 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r45jl" event={"ID":"abf512bb-898b-4547-8286-6219249d80f9","Type":"ContainerStarted","Data":"b92e1e44fb97840343ec061b2b1487bf33be99820b46ad2796d102f3e7fa3089"} Jul 22 02:28:32.387713 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.387687 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"c5304b79c8beb6324d427d3cccd90f3e6f032b3944f5f1309a34dccad1ce9327"} Jul 22 02:28:32.388724 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.388706 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dctdr" event={"ID":"95b795f1-87af-4837-b0c7-6c2d701f34e3","Type":"ContainerStarted","Data":"b8fe90112607bcda6b7e7a47a68b460da870ab2ea04597e2da828648dffc7966"} Jul 22 02:28:32.849873 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.849645 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:32.849873 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.849854 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:32.850060 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.849884 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:32.850060 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.849962 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.84994198 +0000 UTC m=+6.006954247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:32.850060 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.850007 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:32.850060 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.850056 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.850041072 +0000 UTC m=+6.007053339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:32.951766 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:32.951048 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:32.951766 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.951240 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:32.951766 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.951260 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:32.951766 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.951272 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:32.951766 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:32.951329 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.951309534 +0000 UTC m=+6.108321798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:33.295254 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.295134 2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:23:31 +0000 UTC" deadline="2027-12-22 12:49:19.320210548 +0000 UTC" Jul 22 02:28:33.295254 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.295202 2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12442h20m46.025012931s" Jul 22 02:28:33.376458 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.375643 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:33.376458 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:33.375773 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:33.376458 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.376232 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:33.376458 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:33.376338 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:33.396813 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.396185 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" event={"ID":"5786553615a694db0fe10013f4d054ba","Type":"ContainerStarted","Data":"8d25d1106f60a2ab5367a8800aac2aca58d472f3218415c2d47ef3e2c1e92bd5"} Jul 22 02:28:33.406205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.405620 2549 generic.go:356] "Generic (PLEG): container finished" podID="d5c509b29f0f55da12efeeb271236c0a" containerID="64e6f8148364c07a79b60527c1856b6920b1546a915d11bd902cbb5c4b8807a1" exitCode=0 Jul 22 02:28:33.406205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.405671 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"d5c509b29f0f55da12efeeb271236c0a","Type":"ContainerDied","Data":"64e6f8148364c07a79b60527c1856b6920b1546a915d11bd902cbb5c4b8807a1"} Jul 22 02:28:33.428931 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:33.428874 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-125.ec2.internal" podStartSLOduration=3.428853677 podStartE2EDuration="3.428853677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:33.412727225 +0000 UTC m=+4.569739508" watchObservedRunningTime="2026-07-22 02:28:33.428853677 +0000 UTC m=+4.585865962" Jul 22 02:28:34.372666 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.372628 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:34.373116 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.372759 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:34.438287 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.437718 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" event={"ID":"d5c509b29f0f55da12efeeb271236c0a","Type":"ContainerStarted","Data":"ae4c9ec08a0046dc9e531bab842911fd7a408c2dc14f6bec8eef8162ba2ecdaf"} Jul 22 02:28:34.455406 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.455347 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-125.ec2.internal" podStartSLOduration=4.455328609 podStartE2EDuration="4.455328609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:34.454858097 +0000 UTC m=+5.611870382" watchObservedRunningTime="2026-07-22 02:28:34.455328609 +0000 UTC m=+5.612340894" Jul 22 02:28:34.868425 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.868319 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:34.868425 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.868384 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:34.868650 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.868520 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:34.868650 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.868577 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:38.868559098 +0000 UTC m=+10.025571361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:34.868650 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.868643 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:34.868788 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.868677 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:38.868665537 +0000 UTC m=+10.025677798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:34.969799 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:34.969764 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:34.969959 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.969926 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:34.969959 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.969946 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:34.970080 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.970008 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:34.970080 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:34.970064 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:28:38.970046111 +0000 UTC m=+10.127058375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:35.373896 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:35.373385 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:35.373896 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:35.373400 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:35.373896 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:35.373545 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:35.373896 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:35.373781 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:36.373092 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:36.372574 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:36.373092 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:36.372721 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:37.375917 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:37.375888 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:37.376363 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:37.376034 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:37.376464 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:37.376369 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:37.376520 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:37.376462 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:38.372957 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:38.372929 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:38.373123 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:38.373062 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:38.905903 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:38.905863 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:38.906369 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:38.905917 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:38.906369 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:38.906040 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:38.906369 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:38.906051 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:38.906369 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:38.906098 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:46.906082788 +0000 UTC m=+18.063095052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:38.906369 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:38.906123 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:46.906105227 +0000 UTC m=+18.063117489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:39.006894 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:39.006853 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:39.007087 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.007055 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:39.007155 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.007092 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:39.007155 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.007106 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:39.007262 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.007175 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:28:47.007157039 +0000 UTC m=+18.164169306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:39.374218 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:39.374131 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:39.374461 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.374290 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:39.374461 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:39.374341 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:39.374586 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:39.374534 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:40.372998 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:40.372960 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:40.373478 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:40.373081 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:41.373548 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:41.373513 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:41.374055 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:41.373653 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:41.374055 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:41.374051 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:41.374193 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:41.374152 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:42.372678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:42.372638 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:42.372884 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:42.372789 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:43.372995 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:43.372960 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:43.373471 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:43.372958 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:43.373471 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:43.373098 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:43.373471 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:43.373166 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:44.373534 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:44.373506 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:44.373930 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:44.373615 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:45.373431 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:45.373389 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:45.373431 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:45.373431 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:45.373975 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:45.373568 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:45.373975 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:45.373668 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:46.373358 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:46.373324 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:46.373615 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:46.373464 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:46.969354 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:46.969318 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:46.969354 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:46.969365 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:46.969613 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:46.969495 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:46.969613 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:46.969498 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:46.969613 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:46.969564 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:02.969543107 +0000 UTC m=+34.126555380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:28:46.969613 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:46.969585 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:02.969575266 +0000 UTC m=+34.126587532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:28:47.069741 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:47.069694 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:47.069918 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.069887 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:28:47.069918 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.069911 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:28:47.070005 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.069924 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:47.070005 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.069993 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:29:03.06997343 +0000 UTC m=+34.226985692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:28:47.373578 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:47.373497 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:47.373735 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:47.373504 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:47.373735 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.373604 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:47.373735 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:47.373677 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:48.373353 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:48.373322 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:48.373535 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:48.373437 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:49.374465 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:49.374427 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:49.374465 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:49.374459 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:49.374978 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:49.374560 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:49.375030 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:49.374993 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:50.373198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:50.373169 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:50.373372 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:50.373279 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:51.373304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.372681 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:51.373304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.372855 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:51.373304 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:51.372974 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:51.373304 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:51.373228 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:51.473530 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.473330 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-klzgp" event={"ID":"c434dc6a-0628-43fe-9915-263b2bbd3a64","Type":"ContainerStarted","Data":"7e5d9042fd674fc954a379634a262bb956c87fb3068ab2f103ab2edeb82b45aa"} Jul 22 02:28:51.474668 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.474643 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qbfxt" event={"ID":"50f601a1-dfc7-45a1-9028-0b514b932161","Type":"ContainerStarted","Data":"0ad50537812ca29e27591dd00ded5c26d40b9c78fae2a4b7c2aec462a70ced02"} Jul 22 02:28:51.477150 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.476775 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pp4ls" event={"ID":"bdeb088c-7753-438d-a41d-6bf69688c907","Type":"ContainerStarted","Data":"1f8b03a07bb7b5ccb77a9d10fca671c3bc0260221b8178e3bdfcbb14f3a1dbdc"} Jul 22 02:28:51.478974 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.478922 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j8qnh" event={"ID":"b0a71a19-21a0-4493-b9ae-c8b2c8e9845c","Type":"ContainerStarted","Data":"2b7e57ba3a9809b9177e259e5fe610b384bcaef725dc04302d5702989d1957ad"} Jul 22 02:28:51.480331 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.480309 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerStarted","Data":"84e7637a18d641f4dbaf5db3eb148b80d548b7a15f01ca3343a2741e43a928a1"} Jul 22 02:28:51.482646 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.481839 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" event={"ID":"7ac5ebfc-935d-47f0-9d0d-ba27668f791f","Type":"ContainerStarted","Data":"e7d6c2ee0abea012bc10ac848f6f477aa2009f6242a07ad2d7105625f4ed83a0"} Jul 22 02:28:51.483203 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.483181 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r45jl" event={"ID":"abf512bb-898b-4547-8286-6219249d80f9","Type":"ContainerStarted","Data":"7fb2258ea9a5ca92dd5fc62b2b283b54fc6a1fd1e2d9daed4474bbbc88ebcbab"} Jul 22 02:28:51.485278 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.485257 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"15ae6661f9db68c93f8aec1117e418bd89660dc8cb41263b580cfe4274c5a07e"} Jul 22 02:28:51.494666 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.494585 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-klzgp" podStartSLOduration=3.611057292 podStartE2EDuration="22.494565526s" totalImagesPullingTime="18.883508234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:51.493791167 +0000 UTC m=+22.650803442" watchObservedRunningTime="2026-07-22 02:28:51.494565526 +0000 UTC m=+22.651577811" Jul 22 02:28:51.533236 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.533041 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r45jl" podStartSLOduration=11.708373421 podStartE2EDuration="22.533024678s" totalImagesPullingTime="10.824651257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:51.532713495 +0000 UTC m=+22.689725778" watchObservedRunningTime="2026-07-22 02:28:51.533024678 +0000 UTC m=+22.690036961" Jul 22 02:28:51.569308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.569264 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pp4ls" podStartSLOduration=3.645264359 podStartE2EDuration="22.569248272s" totalImagesPullingTime="18.923983913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:51.551927712 +0000 UTC m=+22.708939994" watchObservedRunningTime="2026-07-22 02:28:51.569248272 +0000 UTC m=+22.726260555" Jul 22 02:28:51.569756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:51.569730 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qbfxt" podStartSLOduration=3.701846722 podStartE2EDuration="22.569722634s" totalImagesPullingTime="18.867875912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:51.569160173 +0000 UTC m=+22.726172456" watchObservedRunningTime="2026-07-22 02:28:51.569722634 +0000 UTC m=+22.726734917" Jul 22 02:28:52.373153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.373115 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:52.373331 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:52.373237 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:52.437678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.437638 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:52.438368 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.438346 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:52.454846 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.454785 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j8qnh" podStartSLOduration=4.594509178 podStartE2EDuration="23.454764789s" totalImagesPullingTime="18.860255611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:51.589570947 +0000 UTC m=+22.746583241" watchObservedRunningTime="2026-07-22 02:28:52.454764789 +0000 UTC m=+23.611777080" Jul 22 02:28:52.490334 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.490296 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"fbbcfecf86e69ce6387387d9b5807da261ca4455db11fcd59759e8df9e73befe"} Jul 22 02:28:52.490534 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.490383 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"e26c54256cfc3a93ccafb779f4ef9f7037cd33d4a06781fe4bea05f5684373cf"} Jul 22 02:28:52.490534 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.490446 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"4d1dda7e90c8e43c69651531ef2f6aa63f28bfab42fe33baf514a0c6859d603d"} Jul 22 02:28:52.490534 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.490489 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"ab30442b8ddea01261ebbde4d837e1ab6ed2e5eb31790e417246ce39700cc288"} Jul 22 02:28:52.490534 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.490531 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"fd6829022df3909751f383ec0922d580876cc6a825a4df008782f23046d0493b"} Jul 22 02:28:52.491586 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.491551 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="84e7637a18d641f4dbaf5db3eb148b80d548b7a15f01ca3343a2741e43a928a1" exitCode=0 Jul 22 02:28:52.491684 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.491658 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"84e7637a18d641f4dbaf5db3eb148b80d548b7a15f01ca3343a2741e43a928a1"} Jul 22 02:28:52.493048 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.492449 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:52.493048 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.492587 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r45jl" Jul 22 02:28:52.738119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:52.738087 2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 02:28:53.288956 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.288859 2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T02:28:52.738105328Z","UUID":"49c3ef87-ed65-42d0-942e-50925a9851f6","Handler":null,"Name":"","Endpoint":""} Jul 22 02:28:53.290664 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.290637 2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 02:28:53.290795 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.290679 2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 02:28:53.374548 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.374515 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:53.375009 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:53.374624 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:53.375009 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.374917 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:53.375009 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:53.374988 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:53.496545 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.496504 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" event={"ID":"7ac5ebfc-935d-47f0-9d0d-ba27668f791f","Type":"ContainerStarted","Data":"87e11ebce8eaafa977620083a73e88716e59dce0964fd609fa60d755427b62f7"} Jul 22 02:28:53.498144 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.498080 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dctdr" event={"ID":"95b795f1-87af-4837-b0c7-6c2d701f34e3","Type":"ContainerStarted","Data":"7cd929274d7b6334285ad7044ead8ecccbd8c464c04699178e8b080e1dfa16d8"} Jul 22 02:28:53.514253 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:53.514198 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dctdr" podStartSLOduration=5.650529675 podStartE2EDuration="24.514180619s" totalImagesPullingTime="18.863650944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:53.514080685 +0000 UTC m=+24.671092967" watchObservedRunningTime="2026-07-22 02:28:53.514180619 +0000 UTC m=+24.671192904" Jul 22 02:28:54.373302 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:54.373219 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:54.373468 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:54.373352 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:54.503026 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:54.502990 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" event={"ID":"7ac5ebfc-935d-47f0-9d0d-ba27668f791f","Type":"ContainerStarted","Data":"a54856501abed11acfa627b406215215ce6c9d5e91d80daf9e58629b8b0364b5"} Jul 22 02:28:54.507385 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:54.507353 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"a69cbfc5bf8a93df30fd00d7a8dac26c2c47370f0b4fbf94df49560516ad4c89"} Jul 22 02:28:54.528784 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:54.528724 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-swhth" podStartSLOduration=4.370962084 podStartE2EDuration="25.528699386s" totalImagesPullingTime="21.157737302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:54.527531587 +0000 UTC m=+25.684543869" watchObservedRunningTime="2026-07-22 02:28:54.528699386 +0000 UTC m=+25.685711672" Jul 22 02:28:55.373802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:55.373608 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:55.373987 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:55.373673 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:55.373987 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:55.373915 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:55.374170 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:55.374146 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:56.373537 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:56.373504 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:56.373927 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:56.373611 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:57.373304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.373122 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:57.373304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.373122 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:57.373559 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:57.373395 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:57.373559 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:57.373485 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:57.516376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.516290 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="57f22377941e8fe971c53ef9c0f764791e0f13449f32e7a2799c260d32ae9625" exitCode=0 Jul 22 02:28:57.516376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.516329 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"57f22377941e8fe971c53ef9c0f764791e0f13449f32e7a2799c260d32ae9625"} Jul 22 02:28:57.520023 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.520001 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" event={"ID":"e6f93fc3-b381-431c-a247-d2c9c4c31f2c","Type":"ContainerStarted","Data":"f83fb96f45c3611e7a25a94bc5aeeb21631c397d91bb5b5856f51ba8b83cc887"} Jul 22 02:28:57.520316 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.520293 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:57.520387 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.520346 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:57.535648 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.535628 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:57.565623 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:57.565577 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" podStartSLOduration=9.59689045 podStartE2EDuration="28.565560838s" totalImagesPullingTime="18.968670388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:57.565029025 +0000 UTC m=+28.722041309" watchObservedRunningTime="2026-07-22 02:28:57.565560838 +0000 UTC m=+28.722573120" Jul 22 02:28:58.373159 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:58.373125 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:28:58.373353 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:58.373228 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:28:58.522984 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:58.522947 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:58.538110 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:58.538080 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:28:59.373909 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:59.373872 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:28:59.374077 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:59.373969 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:28:59.374077 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:59.374022 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:28:59.374144 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:28:59.374106 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:28:59.526640 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:59.526610 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="07f61e76d4308e66ce9f501b0450d3ecc4f81743269dbb827f1ee51eebb99cbb" exitCode=0 Jul 22 02:28:59.527088 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:28:59.526708 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"07f61e76d4308e66ce9f501b0450d3ecc4f81743269dbb827f1ee51eebb99cbb"} Jul 22 02:29:00.373642 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:00.373607 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:00.373810 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:00.373706 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:01.372732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:01.372683 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:01.372732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:01.372722 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:01.373267 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:01.372813 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:01.373267 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:01.372936 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:01.532767 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:01.532736 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="f686625306bf474773e5def11317173fac8afba95d720869971e2c9813dbde47" exitCode=0 Jul 22 02:29:01.532917 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:01.532796 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"f686625306bf474773e5def11317173fac8afba95d720869971e2c9813dbde47"} Jul 22 02:29:02.373093 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:02.373062 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:02.373542 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:02.373182 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:02.988585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:02.988405 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:02.988585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:02.988505 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:02.988585 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:02.988547 2549 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 02:29:02.988881 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:02.988626 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret podName:2a35637e-f5c1-452f-9b43-46f19f25314f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:34.988606365 +0000 UTC m=+66.145618641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret") pod "global-pull-secret-syncer-q6k5b" (UID: "2a35637e-f5c1-452f-9b43-46f19f25314f") : object "kube-system"/"original-pull-secret" not registered Jul 22 02:29:02.988881 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:02.988658 2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:29:02.988881 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:02.988723 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs podName:d191d66b-70ec-41d9-a66c-dc564775ae3f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:34.988708573 +0000 UTC m=+66.145720843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs") pod "network-metrics-daemon-lq5wc" (UID: "d191d66b-70ec-41d9-a66c-dc564775ae3f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 02:29:03.089053 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:03.089018 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:03.089290 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.089247 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 02:29:03.089346 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.089301 2549 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 02:29:03.089346 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.089317 2549 projected.go:196] Error preparing data for projected volume kube-api-access-kz9wk for pod openshift-network-diagnostics/network-check-target-wcgkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:29:03.089451 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.089375 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk podName:5cb12023-a8fd-49d8-83bf-4ba33f04f0ff nodeName:}" failed. No retries permitted until 2026-07-22 02:29:35.089355731 +0000 UTC m=+66.246367994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kz9wk" (UniqueName: "kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk") pod "network-check-target-wcgkq" (UID: "5cb12023-a8fd-49d8-83bf-4ba33f04f0ff") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 02:29:03.372801 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:03.372722 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:03.372962 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.372843 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:03.373140 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:03.372731 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:03.373498 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:03.373227 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:04.373467 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:04.373427 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:04.373916 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:04.373568 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:05.377243 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:05.377204 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:05.377721 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:05.377318 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:05.377721 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:05.377382 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:05.377721 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:05.377521 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:05.545760 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:05.545716 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerStarted","Data":"2ac9d4976540504b73cdad3fbda22ecb58de6b28160f474b4bd1a5d095ad9cd1"} Jul 22 02:29:06.373347 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:06.373308 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:06.373543 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:06.373461 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:06.550791 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:06.550761 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="2ac9d4976540504b73cdad3fbda22ecb58de6b28160f474b4bd1a5d095ad9cd1" exitCode=0 Jul 22 02:29:06.551192 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:06.550804 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"2ac9d4976540504b73cdad3fbda22ecb58de6b28160f474b4bd1a5d095ad9cd1"} Jul 22 02:29:07.269913 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.269648 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6k5b"] Jul 22 02:29:07.270073 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.269995 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:07.270138 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:07.270103 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:07.270700 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.270667 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq5wc"] Jul 22 02:29:07.270815 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.270779 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:07.270927 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:07.270906 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:07.283481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.283445 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wcgkq"] Jul 22 02:29:07.283697 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.283680 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:07.283801 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:07.283781 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:07.558162 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.558095 2549 generic.go:356] "Generic (PLEG): container finished" podID="f8641f99-d34b-4f4e-8af0-22b380077196" containerID="b62c5165ec4f21fcbcea9647776012c80042e9b639da41e1e5d2c77d89eb52c2" exitCode=0 Jul 22 02:29:07.558519 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:07.558156 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerDied","Data":"b62c5165ec4f21fcbcea9647776012c80042e9b639da41e1e5d2c77d89eb52c2"} Jul 22 02:29:08.372772 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:08.372740 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:08.372939 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:08.372842 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:08.372939 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:08.372895 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:08.373007 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:08.372985 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:08.563474 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:08.563437 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" event={"ID":"f8641f99-d34b-4f4e-8af0-22b380077196","Type":"ContainerStarted","Data":"99f529968808380ac0fe7e247895c81166603925fb9e7946ee2c171eed846de3"} Jul 22 02:29:08.594296 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:08.594253 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4cnn" podStartSLOduration=10.060695183 podStartE2EDuration="39.594236991s" totalImagesPullingTime="29.533541808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:08.592922897 +0000 UTC m=+39.749935179" watchObservedRunningTime="2026-07-22 02:29:08.594236991 +0000 UTC m=+39.751249273" Jul 22 02:29:09.375213 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:09.375120 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:09.376947 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:09.376879 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wcgkq" podUID="5cb12023-a8fd-49d8-83bf-4ba33f04f0ff" Jul 22 02:29:10.372820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:10.372783 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:10.373322 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:10.372796 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:10.373322 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:10.372894 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq5wc" podUID="d191d66b-70ec-41d9-a66c-dc564775ae3f" Jul 22 02:29:10.373322 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:10.372970 2549 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q6k5b" podUID="2a35637e-f5c1-452f-9b43-46f19f25314f" Jul 22 02:29:11.204546 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.204514 2549 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-125.ec2.internal" event="NodeReady" Jul 22 02:29:11.204689 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.204659 2549 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 02:29:11.267218 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.267187 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84dd6fdd8-dp7dt"] Jul 22 02:29:11.288616 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.288589 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.291388 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.291353 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 02:29:11.291567 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.291548 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 02:29:11.291669 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.291650 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 02:29:11.292446 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.292430 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-48hxd\"" Jul 22 02:29:11.304229 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.304210 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 02:29:11.305444 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.305407 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qqk89"] Jul 22 02:29:11.323362 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.323333 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd6fdd8-dp7dt"] Jul 22 02:29:11.323502 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.323387 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jpslx"] Jul 22 02:29:11.323502 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.323454 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.329184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.328563 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2lw7w\"" Jul 22 02:29:11.329184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.328927 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 02:29:11.329184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.329082 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 02:29:11.343780 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.343760 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqk89"] Jul 22 02:29:11.343898 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.343806 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpslx"] Jul 22 02:29:11.343898 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.343872 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.348314 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.348282 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 02:29:11.348515 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.348344 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 02:29:11.348588 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.348510 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 02:29:11.348701 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.348590 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-btmmd\"" Jul 22 02:29:11.348701 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.348515 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 02:29:11.372723 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.372700 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:11.377064 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.377042 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 02:29:11.377469 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.377186 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb24x\"" Jul 22 02:29:11.377469 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.377231 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 02:29:11.401216 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.401183 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9wdkj"] Jul 22 02:29:11.416876 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.416850 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wdkj"] Jul 22 02:29:11.417012 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.416942 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.423289 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.423270 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 02:29:11.423289 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.423283 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vvdpl\"" Jul 22 02:29:11.423499 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.423328 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 02:29:11.423499 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.423377 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 02:29:11.448005 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.447970 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-tls\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448005 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448002 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-installation-pull-secrets\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448024 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbeda9-2838-4cf6-aeef-21a682e27f19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448041 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ll8\" (UniqueName: \"kubernetes.io/projected/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-api-access-s9ll8\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448155 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8qv\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-kube-api-access-dw8qv\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448195 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbeda9-2838-4cf6-aeef-21a682e27f19-crio-socket\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448228 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50c672fa-b2ef-473b-8ab5-550546d03174-config-volume\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.448275 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448252 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50c672fa-b2ef-473b-8ab5-550546d03174-metrics-tls\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448273 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c672fa-b2ef-473b-8ab5-550546d03174-tmp-dir\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448343 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbeda9-2838-4cf6-aeef-21a682e27f19-data-volume\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448368 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448397 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-image-registry-private-configuration\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448449 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-bound-sa-token\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448475 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-certificates\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448511 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-ca-trust-extracted\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448580 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-trusted-ca\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.448652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.448611 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxzm\" (UniqueName: \"kubernetes.io/projected/50c672fa-b2ef-473b-8ab5-550546d03174-kube-api-access-9hxzm\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.549258 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549181 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbeda9-2838-4cf6-aeef-21a682e27f19-crio-socket\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.549258 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549217 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50c672fa-b2ef-473b-8ab5-550546d03174-config-volume\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.549258 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549235 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50c672fa-b2ef-473b-8ab5-550546d03174-metrics-tls\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.549258 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549258 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c672fa-b2ef-473b-8ab5-550546d03174-tmp-dir\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.549574 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549381 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbeda9-2838-4cf6-aeef-21a682e27f19-data-volume\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.549574 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549388 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6efbeda9-2838-4cf6-aeef-21a682e27f19-crio-socket\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.549574 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549441 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.549574 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549553 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97f5f013-8a28-4018-a9ce-92aaba038aad-cert\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549592 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-image-registry-private-configuration\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549621 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hl8\" (UniqueName: \"kubernetes.io/projected/97f5f013-8a28-4018-a9ce-92aaba038aad-kube-api-access-p5hl8\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549650 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-bound-sa-token\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549680 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-certificates\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549703 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-ca-trust-extracted\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549730 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6efbeda9-2838-4cf6-aeef-21a682e27f19-data-volume\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.549746 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549740 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-trusted-ca\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549762 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxzm\" (UniqueName: \"kubernetes.io/projected/50c672fa-b2ef-473b-8ab5-550546d03174-kube-api-access-9hxzm\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549837 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-tls\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549860 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-installation-pull-secrets\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549881 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbeda9-2838-4cf6-aeef-21a682e27f19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549900 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ll8\" (UniqueName: \"kubernetes.io/projected/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-api-access-s9ll8\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549934 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8qv\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-kube-api-access-dw8qv\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.550058 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-ca-trust-extracted\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.550067 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.550065 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.550447 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.550093 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50c672fa-b2ef-473b-8ab5-550546d03174-config-volume\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.550447 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.549700 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c672fa-b2ef-473b-8ab5-550546d03174-tmp-dir\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.551054 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.550851 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-certificates\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.551154 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.551029 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-trusted-ca\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.553700 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.553676 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50c672fa-b2ef-473b-8ab5-550546d03174-metrics-tls\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.553783 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.553763 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6efbeda9-2838-4cf6-aeef-21a682e27f19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.553841 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.553791 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-image-registry-private-configuration\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.553841 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.553799 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-installation-pull-secrets\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.554026 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.554006 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-registry-tls\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.563427 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.563386 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-bound-sa-token\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.564159 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.564140 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ll8\" (UniqueName: \"kubernetes.io/projected/6efbeda9-2838-4cf6-aeef-21a682e27f19-kube-api-access-s9ll8\") pod \"insights-runtime-extractor-jpslx\" (UID: \"6efbeda9-2838-4cf6-aeef-21a682e27f19\") " pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.565244 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.565227 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxzm\" (UniqueName: \"kubernetes.io/projected/50c672fa-b2ef-473b-8ab5-550546d03174-kube-api-access-9hxzm\") pod \"dns-default-qqk89\" (UID: \"50c672fa-b2ef-473b-8ab5-550546d03174\") " pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.576597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.576576 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8qv\" (UniqueName: \"kubernetes.io/projected/ea3c4fa3-844c-4bbf-853b-d4740a2d0576-kube-api-access-dw8qv\") pod \"image-registry-84dd6fdd8-dp7dt\" (UID: \"ea3c4fa3-844c-4bbf-853b-d4740a2d0576\") " pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.598101 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.598073 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:11.635096 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.635066 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:11.650390 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.650364 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97f5f013-8a28-4018-a9ce-92aaba038aad-cert\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.650390 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.650392 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hl8\" (UniqueName: \"kubernetes.io/projected/97f5f013-8a28-4018-a9ce-92aaba038aad-kube-api-access-p5hl8\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.651841 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.651819 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jpslx" Jul 22 02:29:11.652706 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.652686 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/97f5f013-8a28-4018-a9ce-92aaba038aad-cert\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.663963 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.663939 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hl8\" (UniqueName: \"kubernetes.io/projected/97f5f013-8a28-4018-a9ce-92aaba038aad-kube-api-access-p5hl8\") pod \"ingress-canary-9wdkj\" (UID: \"97f5f013-8a28-4018-a9ce-92aaba038aad\") " pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.725192 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.724807 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9wdkj" Jul 22 02:29:11.825531 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.820625 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd6fdd8-dp7dt"] Jul 22 02:29:11.831525 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:11.831494 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3c4fa3_844c_4bbf_853b_d4740a2d0576.slice/crio-6e3f1173ce161bf0d395a4d42517a082677ede9daab43994d66c60dbf15b1662 WatchSource:0}: Error finding container 6e3f1173ce161bf0d395a4d42517a082677ede9daab43994d66c60dbf15b1662: Status 404 returned error can't find the container with id 6e3f1173ce161bf0d395a4d42517a082677ede9daab43994d66c60dbf15b1662 Jul 22 02:29:11.844955 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.844926 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqk89"] Jul 22 02:29:11.848255 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.848176 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jpslx"] Jul 22 02:29:11.848967 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:11.848937 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c672fa_b2ef_473b_8ab5_550546d03174.slice/crio-ae970900924aeeef8ba6f81c15c4a0bad6a8e712627cfa49189db773a422f307 WatchSource:0}: Error finding container ae970900924aeeef8ba6f81c15c4a0bad6a8e712627cfa49189db773a422f307: Status 404 returned error can't find the container with id ae970900924aeeef8ba6f81c15c4a0bad6a8e712627cfa49189db773a422f307 Jul 22 02:29:11.852106 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:11.852080 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efbeda9_2838_4cf6_aeef_21a682e27f19.slice/crio-208bba322b0a677a2f8df8864541ddb004dcc5063c37ddee93ab11ca820a73b7 WatchSource:0}: Error finding container 208bba322b0a677a2f8df8864541ddb004dcc5063c37ddee93ab11ca820a73b7: Status 404 returned error can't find the container with id 208bba322b0a677a2f8df8864541ddb004dcc5063c37ddee93ab11ca820a73b7 Jul 22 02:29:11.894894 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:11.894868 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9wdkj"] Jul 22 02:29:11.898228 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:11.898208 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f5f013_8a28_4018_a9ce_92aaba038aad.slice/crio-b90f740f01f4660fd84bf672c7f9a8ea8ef660ab0e580523896502f149458764 WatchSource:0}: Error finding container b90f740f01f4660fd84bf672c7f9a8ea8ef660ab0e580523896502f149458764: Status 404 returned error can't find the container with id b90f740f01f4660fd84bf672c7f9a8ea8ef660ab0e580523896502f149458764 Jul 22 02:29:12.374551 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.374512 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:12.375033 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.375009 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:12.377930 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.377910 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:29:12.378368 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.378004 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:29:12.378625 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.378603 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-slk65\"" Jul 22 02:29:12.580337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.579595 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqk89" event={"ID":"50c672fa-b2ef-473b-8ab5-550546d03174","Type":"ContainerStarted","Data":"ae970900924aeeef8ba6f81c15c4a0bad6a8e712627cfa49189db773a422f307"} Jul 22 02:29:12.581579 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.581490 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wdkj" event={"ID":"97f5f013-8a28-4018-a9ce-92aaba038aad","Type":"ContainerStarted","Data":"b90f740f01f4660fd84bf672c7f9a8ea8ef660ab0e580523896502f149458764"} Jul 22 02:29:12.583050 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.582968 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpslx" event={"ID":"6efbeda9-2838-4cf6-aeef-21a682e27f19","Type":"ContainerStarted","Data":"208bba322b0a677a2f8df8864541ddb004dcc5063c37ddee93ab11ca820a73b7"} Jul 22 02:29:12.587217 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.586270 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" event={"ID":"ea3c4fa3-844c-4bbf-853b-d4740a2d0576","Type":"ContainerStarted","Data":"876d272d75f58586bfe401045b4e950e73bb95f5c3f60af1ce3c8e43e478428c"} Jul 22 02:29:12.587217 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.586315 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" event={"ID":"ea3c4fa3-844c-4bbf-853b-d4740a2d0576","Type":"ContainerStarted","Data":"6e3f1173ce161bf0d395a4d42517a082677ede9daab43994d66c60dbf15b1662"} Jul 22 02:29:12.587217 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.587156 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:12.615212 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:12.614500 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" podStartSLOduration=7.614479199 podStartE2EDuration="7.614479199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:12.613190705 +0000 UTC m=+43.770202989" watchObservedRunningTime="2026-07-22 02:29:12.614479199 +0000 UTC m=+43.771491483" Jul 22 02:29:19.609754 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.609513 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqk89" event={"ID":"50c672fa-b2ef-473b-8ab5-550546d03174","Type":"ContainerStarted","Data":"72f074e6e0a0ee6c876246bf8efb2fa552398f5d2e3f9a159cd551638fa9e519"} Jul 22 02:29:19.610642 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.609787 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqk89" event={"ID":"50c672fa-b2ef-473b-8ab5-550546d03174","Type":"ContainerStarted","Data":"80410d954e0069c3cd90382d1d5f8ecf499ec13feb1c45a515d87260b8f362f3"} Jul 22 02:29:19.610642 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.609823 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:19.611293 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.611269 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9wdkj" event={"ID":"97f5f013-8a28-4018-a9ce-92aaba038aad","Type":"ContainerStarted","Data":"a92f13ea6ae4a9cd5503dfe540400798a63aef33af16b03a4de6097e72c33d8a"} Jul 22 02:29:19.612861 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.612842 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpslx" event={"ID":"6efbeda9-2838-4cf6-aeef-21a682e27f19","Type":"ContainerStarted","Data":"6513e7c3e53dc7f60af6cceaedf71c0f2d8050f50d85cfbe59c93fc7bc455b81"} Jul 22 02:29:19.640710 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.640647 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qqk89" podStartSLOduration=1.756635128 podStartE2EDuration="8.640627808s" totalImagesPullingTime="6.88399268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:19.639382213 +0000 UTC m=+50.796394510" watchObservedRunningTime="2026-07-22 02:29:19.640627808 +0000 UTC m=+50.797640092" Jul 22 02:29:19.668094 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:19.668022 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9wdkj" podStartSLOduration=1.804499957 podStartE2EDuration="8.66799652s" totalImagesPullingTime="6.863496563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:19.666706475 +0000 UTC m=+50.823718763" watchObservedRunningTime="2026-07-22 02:29:19.66799652 +0000 UTC m=+50.825008804" Jul 22 02:29:23.625373 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:23.625146 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpslx" event={"ID":"6efbeda9-2838-4cf6-aeef-21a682e27f19","Type":"ContainerStarted","Data":"b7db55ca278155fa1bc64fc8264a88177813ecd87974b8a16b18fa5ca7ed1cde"} Jul 22 02:29:27.642815 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:27.642773 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jpslx" event={"ID":"6efbeda9-2838-4cf6-aeef-21a682e27f19","Type":"ContainerStarted","Data":"c3be80bb8169bcff957073b369a7946f29ece8f9f860d6b7ef2680a6cd8ca529"} Jul 22 02:29:27.661399 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:27.661353 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jpslx" podStartSLOduration=1.543283331 podStartE2EDuration="16.661309488s" totalImagesPullingTime="15.118026157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:27.660801723 +0000 UTC m=+58.817814007" watchObservedRunningTime="2026-07-22 02:29:27.661309488 +0000 UTC m=+58.818321770" Jul 22 02:29:28.469808 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.469772 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb"] Jul 22 02:29:28.472103 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.472087 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.474716 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.474695 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 02:29:28.474849 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.474729 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 02:29:28.475842 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.475822 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 02:29:28.475906 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.475864 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 02:29:28.475986 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.475967 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 02:29:28.476183 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.476170 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ffbr5\"" Jul 22 02:29:28.480633 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.480613 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1642c6-511d-4287-9919-e5c97f63a516-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.480768 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.480747 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfxqb\" (UniqueName: \"kubernetes.io/projected/bc1642c6-511d-4287-9919-e5c97f63a516-kube-api-access-bfxqb\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.480818 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.480788 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.480871 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.480822 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.481641 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.481624 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-r9972"] Jul 22 02:29:28.483612 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.483599 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.486845 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.486819 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb"] Jul 22 02:29:28.489781 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.489755 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 02:29:28.490448 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.489785 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 02:29:28.490448 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.489866 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tdvp8\"" Jul 22 02:29:28.490448 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.490136 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 02:29:28.503478 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.503449 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-r9972"] Jul 22 02:29:28.534079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.534049 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m7dqn"] Jul 22 02:29:28.536110 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.536091 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.538607 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.538589 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 02:29:28.538961 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.538796 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 02:29:28.539073 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.539053 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z2bb6\"" Jul 22 02:29:28.539354 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.539338 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 02:29:28.581581 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581550 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.581581 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581584 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-textfile\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581775 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581602 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zdms\" (UniqueName: \"kubernetes.io/projected/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-kube-api-access-7zdms\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581775 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581620 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.581775 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581662 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581781 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581800 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfxqb\" (UniqueName: \"kubernetes.io/projected/bc1642c6-511d-4287-9919-e5c97f63a516-kube-api-access-bfxqb\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581813 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-wtmp\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581826 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-metrics-client-ca\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581841 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581857 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.581895 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581880 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhtn\" (UniqueName: \"kubernetes.io/projected/3f856281-f75d-4bad-af78-4a0fbc449557-kube-api-access-rfhtn\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581901 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-root\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581927 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f856281-f75d-4bad-af78-4a0fbc449557-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581954 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1642c6-511d-4287-9919-e5c97f63a516-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.581978 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.582004 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.582026 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.582184 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.582068 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-sys\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.582610 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.582591 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc1642c6-511d-4287-9919-e5c97f63a516-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.585709 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.585683 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.585709 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.585694 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc1642c6-511d-4287-9919-e5c97f63a516-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.592472 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.592450 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfxqb\" (UniqueName: \"kubernetes.io/projected/bc1642c6-511d-4287-9919-e5c97f63a516-kube-api-access-bfxqb\") pod \"openshift-state-metrics-79f6958cdf-ws4pb\" (UID: \"bc1642c6-511d-4287-9919-e5c97f63a516\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.682840 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.682811 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.682840 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.682843 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhtn\" (UniqueName: \"kubernetes.io/projected/3f856281-f75d-4bad-af78-4a0fbc449557-kube-api-access-rfhtn\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.682859 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-root\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.682878 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f856281-f75d-4bad-af78-4a0fbc449557-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:28.682955 2549 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.682960 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-root\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:28.683012 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls podName:6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b nodeName:}" failed. No retries permitted until 2026-07-22 02:29:29.18299263 +0000 UTC m=+60.340004903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls") pod "node-exporter-m7dqn" (UID: "6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b") : secret "node-exporter-tls" not found Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683030 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683070 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683098 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683126 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-sys\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683176 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-textfile\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683203 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3f856281-f75d-4bad-af78-4a0fbc449557-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683204 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zdms\" (UniqueName: \"kubernetes.io/projected/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-kube-api-access-7zdms\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:28.683233 2549 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683239 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-sys\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683251 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.683317 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:29:28.683299 2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls podName:3f856281-f75d-4bad-af78-4a0fbc449557 nodeName:}" failed. No retries permitted until 2026-07-22 02:29:29.183281401 +0000 UTC m=+60.340293663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-r9972" (UID: "3f856281-f75d-4bad-af78-4a0fbc449557") : secret "kube-state-metrics-tls" not found Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683320 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683494 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-wtmp\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683521 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-metrics-client-ca\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683548 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683589 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-textfile\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683676 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-accelerators-collector-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683719 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-wtmp\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.683920 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.684265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.684187 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-metrics-client-ca\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.684989 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.684303 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f856281-f75d-4bad-af78-4a0fbc449557-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.685940 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.685919 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.686532 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.686496 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.691615 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.691594 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zdms\" (UniqueName: \"kubernetes.io/projected/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-kube-api-access-7zdms\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:28.694969 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.694949 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhtn\" (UniqueName: \"kubernetes.io/projected/3f856281-f75d-4bad-af78-4a0fbc449557-kube-api-access-rfhtn\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:28.781279 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.781184 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" Jul 22 02:29:28.905857 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:28.905825 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb"] Jul 22 02:29:28.909475 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:28.909449 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1642c6_511d_4287_9919_e5c97f63a516.slice/crio-a0d3fda2bfa7bc1c1e98f6d3770dc284e26accc83c75c35407a2b1f0184914bc WatchSource:0}: Error finding container a0d3fda2bfa7bc1c1e98f6d3770dc284e26accc83c75c35407a2b1f0184914bc: Status 404 returned error can't find the container with id a0d3fda2bfa7bc1c1e98f6d3770dc284e26accc83c75c35407a2b1f0184914bc Jul 22 02:29:29.187520 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.187472 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:29.187703 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.187551 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:29.190014 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.189991 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b-node-exporter-tls\") pod \"node-exporter-m7dqn\" (UID: \"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b\") " pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:29.190088 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.190063 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f856281-f75d-4bad-af78-4a0fbc449557-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-r9972\" (UID: \"3f856281-f75d-4bad-af78-4a0fbc449557\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:29.399286 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.399205 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tdvp8\"" Jul 22 02:29:29.407058 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.407029 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" Jul 22 02:29:29.447863 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.447834 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z2bb6\"" Jul 22 02:29:29.456027 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.455987 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m7dqn" Jul 22 02:29:29.467169 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:29.467144 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8b6f0b_04f8_45d4_a13c_d1c77fb5843b.slice/crio-d061ee6dd0edb523614fce749588cc6ba695d4ae5f9a5333e18c0ebd7a7b0a02 WatchSource:0}: Error finding container d061ee6dd0edb523614fce749588cc6ba695d4ae5f9a5333e18c0ebd7a7b0a02: Status 404 returned error can't find the container with id d061ee6dd0edb523614fce749588cc6ba695d4ae5f9a5333e18c0ebd7a7b0a02 Jul 22 02:29:29.569982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.569811 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-r9972"] Jul 22 02:29:29.572392 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:29.572359 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f856281_f75d_4bad_af78_4a0fbc449557.slice/crio-138c2df2ec5d821e92ece9610c44871d39e566f5b543e913d639d1c9d3be782b WatchSource:0}: Error finding container 138c2df2ec5d821e92ece9610c44871d39e566f5b543e913d639d1c9d3be782b: Status 404 returned error can't find the container with id 138c2df2ec5d821e92ece9610c44871d39e566f5b543e913d639d1c9d3be782b Jul 22 02:29:29.618394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.618366 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qqk89" Jul 22 02:29:29.651309 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.651223 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7dqn" event={"ID":"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b","Type":"ContainerStarted","Data":"d061ee6dd0edb523614fce749588cc6ba695d4ae5f9a5333e18c0ebd7a7b0a02"} Jul 22 02:29:29.653447 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.653401 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" event={"ID":"bc1642c6-511d-4287-9919-e5c97f63a516","Type":"ContainerStarted","Data":"2088b23d6ac254ea045615351cfb14b95f6284b16f60f77a378d83837c3fa036"} Jul 22 02:29:29.653565 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.653477 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" event={"ID":"bc1642c6-511d-4287-9919-e5c97f63a516","Type":"ContainerStarted","Data":"ea8792eaf96d1287acdab016a621814b8f95f4de26f1990e47b69b1265339780"} Jul 22 02:29:29.653565 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.653508 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" event={"ID":"bc1642c6-511d-4287-9919-e5c97f63a516","Type":"ContainerStarted","Data":"a0d3fda2bfa7bc1c1e98f6d3770dc284e26accc83c75c35407a2b1f0184914bc"} Jul 22 02:29:29.654661 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:29.654642 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" event={"ID":"3f856281-f75d-4bad-af78-4a0fbc449557","Type":"ContainerStarted","Data":"138c2df2ec5d821e92ece9610c44871d39e566f5b543e913d639d1c9d3be782b"} Jul 22 02:29:30.541759 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:30.541729 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2xxp6" Jul 22 02:29:30.660062 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:30.659940 2549 generic.go:356] "Generic (PLEG): container finished" podID="6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b" containerID="7675e194a0343bd04eca68b8cf5b9bdb4c69d3941ee7f866957ecf321ec15dbb" exitCode=0 Jul 22 02:29:30.660062 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:30.660033 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7dqn" event={"ID":"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b","Type":"ContainerDied","Data":"7675e194a0343bd04eca68b8cf5b9bdb4c69d3941ee7f866957ecf321ec15dbb"} Jul 22 02:29:30.662998 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:30.662965 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" event={"ID":"bc1642c6-511d-4287-9919-e5c97f63a516","Type":"ContainerStarted","Data":"b7e72ef452e0270bd662873d3615700d41354cbcb6c799915e0a8176a0649563"} Jul 22 02:29:30.703313 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:30.703250 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-ws4pb" podStartSLOduration=1.716235684 podStartE2EDuration="2.703229478s" totalImagesPullingTime="986.993794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:30.701618006 +0000 UTC m=+61.858630290" watchObservedRunningTime="2026-07-22 02:29:30.703229478 +0000 UTC m=+61.860241762" Jul 22 02:29:31.554351 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.553880 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74c59d48c5-rngbl"] Jul 22 02:29:31.578517 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.578488 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74c59d48c5-rngbl"] Jul 22 02:29:31.578672 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.578610 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.581598 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.581576 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6dm64\"" Jul 22 02:29:31.581779 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.581752 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 02:29:31.581779 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.581753 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 02:29:31.581982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.581805 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-e2qnscd77gf3\"" Jul 22 02:29:31.581982 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.581813 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 02:29:31.582089 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.582072 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 02:29:31.582449 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.582437 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 02:29:31.607718 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607689 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmn5\" (UniqueName: \"kubernetes.io/projected/a9862596-c307-4d89-a162-2d649da6c210-kube-api-access-vcmn5\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.607876 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607765 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9862596-c307-4d89-a162-2d649da6c210-metrics-client-ca\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.607876 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607810 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.607876 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607830 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.607876 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607863 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-grpc-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.608048 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607889 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.608048 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607916 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.608048 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.607935 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.668499 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.668462 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7dqn" event={"ID":"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b","Type":"ContainerStarted","Data":"79009deefcff29aa526eef631dfced65f6be47c7bff3a7f4f9dbd3b7ec978d7f"} Jul 22 02:29:31.668674 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.668528 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m7dqn" event={"ID":"6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b","Type":"ContainerStarted","Data":"ea0a139920f87fb7c342ddece2ff33d49cbbe37b6c1076d5f6b54051007153c5"} Jul 22 02:29:31.708676 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708640 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.708676 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708680 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.708913 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708703 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-grpc-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.708913 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708729 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.708913 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708755 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.709077 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708934 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.709077 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.708994 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmn5\" (UniqueName: \"kubernetes.io/projected/a9862596-c307-4d89-a162-2d649da6c210-kube-api-access-vcmn5\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.709172 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.709154 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9862596-c307-4d89-a162-2d649da6c210-metrics-client-ca\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.710070 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.710034 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9862596-c307-4d89-a162-2d649da6c210-metrics-client-ca\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.711733 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.711710 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.711992 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.711973 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.712054 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.712024 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.712105 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.712027 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.712321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.712303 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.712439 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.712400 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9862596-c307-4d89-a162-2d649da6c210-secret-grpc-tls\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.719518 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.719495 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmn5\" (UniqueName: \"kubernetes.io/projected/a9862596-c307-4d89-a162-2d649da6c210-kube-api-access-vcmn5\") pod \"thanos-querier-74c59d48c5-rngbl\" (UID: \"a9862596-c307-4d89-a162-2d649da6c210\") " pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:31.887225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:31.887197 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:32.089132 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.089085 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m7dqn" podStartSLOduration=3.177695375 podStartE2EDuration="4.089062494s" totalImagesPullingTime="840.352948ms" totalInitContainerRuntime="71.014171ms" isStatefulPod=true podCreationTimestamp="2026-07-22 02:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:31.689047526 +0000 UTC m=+62.846059809" watchObservedRunningTime="2026-07-22 02:29:32.089062494 +0000 UTC m=+63.246074777" Jul 22 02:29:32.089517 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.089495 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74c59d48c5-rngbl"] Jul 22 02:29:32.093864 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:32.093838 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9862596_c307_4d89_a162_2d649da6c210.slice/crio-601270674ec1ffb03e7e5cd4d9fba57a263a989a396e71688fde8a33ca45bc6d WatchSource:0}: Error finding container 601270674ec1ffb03e7e5cd4d9fba57a263a989a396e71688fde8a33ca45bc6d: Status 404 returned error can't find the container with id 601270674ec1ffb03e7e5cd4d9fba57a263a989a396e71688fde8a33ca45bc6d Jul 22 02:29:32.673939 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.673900 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" event={"ID":"3f856281-f75d-4bad-af78-4a0fbc449557","Type":"ContainerStarted","Data":"db07580fda6d41b5f6a0c2b1ddb5eb151d13da4b383754b86be66e5ff386b1fc"} Jul 22 02:29:32.674509 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.673965 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" event={"ID":"3f856281-f75d-4bad-af78-4a0fbc449557","Type":"ContainerStarted","Data":"f385d1201e5e8217cc71acd595861b328048cbe29a5bc14819a587b7285f2a6c"} Jul 22 02:29:32.674509 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.673993 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" event={"ID":"3f856281-f75d-4bad-af78-4a0fbc449557","Type":"ContainerStarted","Data":"5ce3fcb28724a3568bcd64e247e1a409e1580d275f170602ebc35f1ae846b150"} Jul 22 02:29:32.675134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.675115 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"601270674ec1ffb03e7e5cd4d9fba57a263a989a396e71688fde8a33ca45bc6d"} Jul 22 02:29:32.694678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.694631 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-r9972" podStartSLOduration=2.405604696 podStartE2EDuration="4.694617715s" totalImagesPullingTime="2.289013019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:32.693650052 +0000 UTC m=+63.850662334" watchObservedRunningTime="2026-07-22 02:29:32.694617715 +0000 UTC m=+63.851629998" Jul 22 02:29:32.777069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.777035 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7657bfd955-8vx2n"] Jul 22 02:29:32.779212 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.779193 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.781971 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.781941 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 02:29:32.782079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.781947 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 02:29:32.782079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.781979 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fufn2a7e4ds3k\"" Jul 22 02:29:32.782079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.781983 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8m82j\"" Jul 22 02:29:32.782079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.782048 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 02:29:32.782079 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.781953 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 02:29:32.791648 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.791626 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7657bfd955-8vx2n"] Jul 22 02:29:32.818709 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818677 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-tls\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818709 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818718 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-client-certs\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818752 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6480fc-238b-4942-b5f3-6d6b9ad80828-audit-log\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818799 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818815 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-metrics-server-audit-profiles\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56ns\" (UniqueName: \"kubernetes.io/projected/eb6480fc-238b-4942-b5f3-6d6b9ad80828-kube-api-access-r56ns\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.818870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.818860 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-client-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920268 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920236 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-tls\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920268 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920276 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-client-certs\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920461 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6480fc-238b-4942-b5f3-6d6b9ad80828-audit-log\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920518 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920540 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-metrics-server-audit-profiles\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920717 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920562 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56ns\" (UniqueName: \"kubernetes.io/projected/eb6480fc-238b-4942-b5f3-6d6b9ad80828-kube-api-access-r56ns\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.920717 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.920592 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-client-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.921627 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.921376 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eb6480fc-238b-4942-b5f3-6d6b9ad80828-audit-log\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.921627 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.921577 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.921891 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.921870 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eb6480fc-238b-4942-b5f3-6d6b9ad80828-metrics-server-audit-profiles\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.923595 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.923572 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-client-certs\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.924013 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.923968 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-client-ca-bundle\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.924013 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.923986 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eb6480fc-238b-4942-b5f3-6d6b9ad80828-secret-metrics-server-tls\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:32.930634 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:32.930617 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56ns\" (UniqueName: \"kubernetes.io/projected/eb6480fc-238b-4942-b5f3-6d6b9ad80828-kube-api-access-r56ns\") pod \"metrics-server-7657bfd955-8vx2n\" (UID: \"eb6480fc-238b-4942-b5f3-6d6b9ad80828\") " pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:33.089264 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.089226 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:33.235821 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.235790 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7657bfd955-8vx2n"] Jul 22 02:29:33.239018 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:33.238989 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6480fc_238b_4942_b5f3_6d6b9ad80828.slice/crio-4d61ccccb711e9017470fa8c50c80b01ae1ab9b9648118a51f58d7ec3ffe46b2 WatchSource:0}: Error finding container 4d61ccccb711e9017470fa8c50c80b01ae1ab9b9648118a51f58d7ec3ffe46b2: Status 404 returned error can't find the container with id 4d61ccccb711e9017470fa8c50c80b01ae1ab9b9648118a51f58d7ec3ffe46b2 Jul 22 02:29:33.679187 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.679157 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" event={"ID":"eb6480fc-238b-4942-b5f3-6d6b9ad80828","Type":"ContainerStarted","Data":"4d61ccccb711e9017470fa8c50c80b01ae1ab9b9648118a51f58d7ec3ffe46b2"} Jul 22 02:29:33.769631 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.769597 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-86856444f-skxsq"] Jul 22 02:29:33.772529 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.772506 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.774788 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.774770 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 22 02:29:33.774890 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.774795 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 22 02:29:33.775300 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.775283 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 22 02:29:33.775389 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.775378 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 22 02:29:33.776485 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.776468 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 22 02:29:33.777180 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.777161 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7tt4x\"" Jul 22 02:29:33.781266 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.781248 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 22 02:29:33.786058 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.786040 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86856444f-skxsq"] Jul 22 02:29:33.827400 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827359 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827400 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827403 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-federate-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827650 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827437 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-metrics-client-ca\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827650 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827594 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827650 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827641 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827669 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlbgk\" (UniqueName: \"kubernetes.io/projected/371141c7-71f4-4f45-95e2-4c623758459c-kube-api-access-rlbgk\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827701 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-serving-certs-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.827802 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.827769 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928521 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928483 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928521 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928525 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlbgk\" (UniqueName: \"kubernetes.io/projected/371141c7-71f4-4f45-95e2-4c623758459c-kube-api-access-rlbgk\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928544 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-serving-certs-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928591 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928628 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928652 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-federate-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928674 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-metrics-client-ca\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.928751 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.928745 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.930161 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.930081 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.930282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.930211 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-metrics-client-ca\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.931121 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.931097 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/371141c7-71f4-4f45-95e2-4c623758459c-serving-certs-ca-bundle\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.932105 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.932061 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.932673 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.932628 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.932794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.932774 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-federate-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.934074 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.934051 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/371141c7-71f4-4f45-95e2-4c623758459c-telemeter-client-tls\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:33.940123 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:33.940100 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlbgk\" (UniqueName: \"kubernetes.io/projected/371141c7-71f4-4f45-95e2-4c623758459c-kube-api-access-rlbgk\") pod \"telemeter-client-86856444f-skxsq\" (UID: \"371141c7-71f4-4f45-95e2-4c623758459c\") " pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:34.082903 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:34.082638 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" Jul 22 02:29:34.597590 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:34.597562 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84dd6fdd8-dp7dt" Jul 22 02:29:35.003638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.003553 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-86856444f-skxsq"] Jul 22 02:29:35.038319 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.038230 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:35.038474 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.038435 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:35.041509 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.041286 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 02:29:35.041615 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.041526 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 02:29:35.050878 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.050848 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a35637e-f5c1-452f-9b43-46f19f25314f-original-pull-secret\") pod \"global-pull-secret-syncer-q6k5b\" (UID: \"2a35637e-f5c1-452f-9b43-46f19f25314f\") " pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:35.051756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.051737 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d191d66b-70ec-41d9-a66c-dc564775ae3f-metrics-certs\") pod \"network-metrics-daemon-lq5wc\" (UID: \"d191d66b-70ec-41d9-a66c-dc564775ae3f\") " pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:35.139263 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.139184 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:35.141564 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.141545 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 02:29:35.152377 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.152357 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 02:29:35.163119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.163099 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz9wk\" (UniqueName: \"kubernetes.io/projected/5cb12023-a8fd-49d8-83bf-4ba33f04f0ff-kube-api-access-kz9wk\") pod \"network-check-target-wcgkq\" (UID: \"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff\") " pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:35.202161 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.202135 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-slk65\"" Jul 22 02:29:35.207037 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.207019 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q6k5b" Jul 22 02:29:35.210090 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.210066 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq5wc" Jul 22 02:29:35.361519 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.361394 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq5wc"] Jul 22 02:29:35.364339 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:35.364316 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd191d66b_70ec_41d9_a66c_dc564775ae3f.slice/crio-464e9d47b8170abea387ff99ed6a7ab3558ac34b7da5f90fe8fbd24be2a6088f WatchSource:0}: Error finding container 464e9d47b8170abea387ff99ed6a7ab3558ac34b7da5f90fe8fbd24be2a6088f: Status 404 returned error can't find the container with id 464e9d47b8170abea387ff99ed6a7ab3558ac34b7da5f90fe8fbd24be2a6088f Jul 22 02:29:35.381355 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.381332 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q6k5b"] Jul 22 02:29:35.383969 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.383943 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb24x\"" Jul 22 02:29:35.384362 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:35.384344 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a35637e_f5c1_452f_9b43_46f19f25314f.slice/crio-cbd5cf543a1a3d4df438536cd7e992ea35703e2c57322083b362d66c9576042c WatchSource:0}: Error finding container cbd5cf543a1a3d4df438536cd7e992ea35703e2c57322083b362d66c9576042c: Status 404 returned error can't find the container with id cbd5cf543a1a3d4df438536cd7e992ea35703e2c57322083b362d66c9576042c Jul 22 02:29:35.391106 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.391023 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:35.550661 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.550633 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wcgkq"] Jul 22 02:29:35.554082 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:29:35.554055 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb12023_a8fd_49d8_83bf_4ba33f04f0ff.slice/crio-c2d0e7c21fca855038aff98b9ed455947437ed9feac972d1062505c6fd2de7c5 WatchSource:0}: Error finding container c2d0e7c21fca855038aff98b9ed455947437ed9feac972d1062505c6fd2de7c5: Status 404 returned error can't find the container with id c2d0e7c21fca855038aff98b9ed455947437ed9feac972d1062505c6fd2de7c5 Jul 22 02:29:35.690251 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.690203 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq5wc" event={"ID":"d191d66b-70ec-41d9-a66c-dc564775ae3f","Type":"ContainerStarted","Data":"464e9d47b8170abea387ff99ed6a7ab3558ac34b7da5f90fe8fbd24be2a6088f"} Jul 22 02:29:35.692226 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.692191 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6k5b" event={"ID":"2a35637e-f5c1-452f-9b43-46f19f25314f","Type":"ContainerStarted","Data":"cbd5cf543a1a3d4df438536cd7e992ea35703e2c57322083b362d66c9576042c"} Jul 22 02:29:35.694792 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.694770 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" event={"ID":"eb6480fc-238b-4942-b5f3-6d6b9ad80828","Type":"ContainerStarted","Data":"cf479037fdbb741098f78d065563f33164908749d46bed5864639b46bfcff61a"} Jul 22 02:29:35.696463 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.696440 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wcgkq" event={"ID":"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff","Type":"ContainerStarted","Data":"c2d0e7c21fca855038aff98b9ed455947437ed9feac972d1062505c6fd2de7c5"} Jul 22 02:29:35.698823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.698794 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"3c7e51bd7221fe4bdd6d5a2d83aa052474c7992c359fc294128a61231bbf71ea"} Jul 22 02:29:35.698932 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.698873 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"7ec904cdccc6930b64fd929239ff9fef5fadb82caf853ccf09fe7a8d8b820ebd"} Jul 22 02:29:35.698932 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.698916 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"ba3b0843ad121513b90f63c26ad16a1fb577cfc9fe40f8a1bc95e0de12154eb1"} Jul 22 02:29:35.703109 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.702890 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" event={"ID":"371141c7-71f4-4f45-95e2-4c623758459c","Type":"ContainerStarted","Data":"9028741ac09f78cc6fbc0cabef732da52e984c55f5a5a3c2bc65f51517ae8f48"} Jul 22 02:29:35.725131 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:35.725085 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" podStartSLOduration=2.104553259 podStartE2EDuration="3.725064885s" totalImagesPullingTime="1.620511626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:35.724641794 +0000 UTC m=+66.881654082" watchObservedRunningTime="2026-07-22 02:29:35.725064885 +0000 UTC m=+66.882077167" Jul 22 02:29:36.712153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:36.712105 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"7685299391f479a369298791fc6f9ecbb4a6bb155542e11244c0cbc0677212b5"} Jul 22 02:29:36.712647 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:36.712199 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"cfb277a181a5a20cb3f2b99031dd428f9442b4e98b951420400ac93af405696a"} Jul 22 02:29:36.712647 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:36.712237 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" event={"ID":"a9862596-c307-4d89-a162-2d649da6c210","Type":"ContainerStarted","Data":"3fa7737f3d0dc9ad2666a94e753e5b0e619dee885408afc23d5c4c5418ac2b8c"} Jul 22 02:29:36.737761 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:36.737231 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" podStartSLOduration=2.116687529 podStartE2EDuration="5.737214787s" totalImagesPullingTime="3.620527258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:36.737091608 +0000 UTC m=+67.894103895" watchObservedRunningTime="2026-07-22 02:29:36.737214787 +0000 UTC m=+67.894227070" Jul 22 02:29:37.719233 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:37.719187 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" event={"ID":"371141c7-71f4-4f45-95e2-4c623758459c","Type":"ContainerStarted","Data":"6fbc5a2a648ce7ada92591ed705d44ba4ee3f37a314afc4dbee5a98c70b02425"} Jul 22 02:29:37.722481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:37.722316 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq5wc" event={"ID":"d191d66b-70ec-41d9-a66c-dc564775ae3f","Type":"ContainerStarted","Data":"0ddb52ebd368384a384479558af4b6a777e0d13ff969315c9ceb3b25d61b0e54"} Jul 22 02:29:37.722481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:37.722367 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq5wc" event={"ID":"d191d66b-70ec-41d9-a66c-dc564775ae3f","Type":"ContainerStarted","Data":"df79c4eb5927e793e87bc3302bf675117210a70d55d4256a7885e2be4a048bcf"} Jul 22 02:29:37.722481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:37.722395 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:37.742449 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:37.742164 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lq5wc" podStartSLOduration=66.808431328 podStartE2EDuration="1m8.74214656s" totalImagesPullingTime="1.933715232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:37.740973614 +0000 UTC m=+68.897985898" watchObservedRunningTime="2026-07-22 02:29:37.74214656 +0000 UTC m=+68.899158843" Jul 22 02:29:41.738460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.738400 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" event={"ID":"371141c7-71f4-4f45-95e2-4c623758459c","Type":"ContainerStarted","Data":"7585c71745629fcd63594178988580fd47a6442e0bc605d7b08d50d32287434e"} Jul 22 02:29:41.738922 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.738494 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" event={"ID":"371141c7-71f4-4f45-95e2-4c623758459c","Type":"ContainerStarted","Data":"f1426da0a9260d945b2d6fa0a819e2629bddf314eb9001f79f90121b16ee0c9a"} Jul 22 02:29:41.739936 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.739912 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q6k5b" event={"ID":"2a35637e-f5c1-452f-9b43-46f19f25314f","Type":"ContainerStarted","Data":"0257933a62c5dc49064e59732a7cc69e653cd6fd78a47c9f81c3462d8fc7d2f4"} Jul 22 02:29:41.741405 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.741385 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wcgkq" event={"ID":"5cb12023-a8fd-49d8-83bf-4ba33f04f0ff","Type":"ContainerStarted","Data":"188b0dda1e351077f3ad2f0d67f48549c1fe18863d03ca0953322f333ded119d"} Jul 22 02:29:41.741532 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.741522 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:29:41.764864 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.764817 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-86856444f-skxsq" podStartSLOduration=3.020792566 podStartE2EDuration="8.764805404s" totalImagesPullingTime="5.744012838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:41.763788686 +0000 UTC m=+72.920801004" watchObservedRunningTime="2026-07-22 02:29:41.764805404 +0000 UTC m=+72.921817686" Jul 22 02:29:41.781082 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.781038 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wcgkq" podStartSLOduration=67.486752073 podStartE2EDuration="1m12.781024173s" totalImagesPullingTime="5.2942721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:41.780310781 +0000 UTC m=+72.937323064" watchObservedRunningTime="2026-07-22 02:29:41.781024173 +0000 UTC m=+72.938036456" Jul 22 02:29:41.798776 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:41.798723 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q6k5b" podStartSLOduration=67.32301806 podStartE2EDuration="1m12.798709358s" totalImagesPullingTime="5.475691298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:41.798471706 +0000 UTC m=+72.955483989" watchObservedRunningTime="2026-07-22 02:29:41.798709358 +0000 UTC m=+72.955721641" Jul 22 02:29:43.732008 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:43.731983 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74c59d48c5-rngbl" Jul 22 02:29:53.089651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:53.089511 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:29:53.089651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:29:53.089593 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:30:12.748552 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:30:12.748522 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wcgkq" Jul 22 02:30:13.094857 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:30:13.094782 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:30:13.098730 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:30:13.098705 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7657bfd955-8vx2n" Jul 22 02:32:04.091551 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.091513 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42"] Jul 22 02:32:04.094372 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.094352 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.096827 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.096804 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 02:32:04.096957 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.096840 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 02:32:04.097646 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.097632 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fblmd\"" Jul 22 02:32:04.103003 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.102981 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42"] Jul 22 02:32:04.230535 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.230029 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.230809 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.230784 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpg6\" (UniqueName: \"kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.231015 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.230997 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.331761 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.331719 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.331964 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.331776 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.331964 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.331806 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpg6\" (UniqueName: \"kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.332191 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.332168 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.332262 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.332172 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.340116 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.340087 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpg6\" (UniqueName: \"kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.403911 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.403823 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:04.537071 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:04.537049 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42"] Jul 22 02:32:04.539513 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:04.539484 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a328b4_f01d_420c_a9dd_7185f30d1251.slice/crio-a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af WatchSource:0}: Error finding container a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af: Status 404 returned error can't find the container with id a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af Jul 22 02:32:05.266923 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:05.266890 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" event={"ID":"50a328b4-f01d-420c-a9dd-7185f30d1251","Type":"ContainerStarted","Data":"a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af"} Jul 22 02:32:12.294835 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:12.294802 2549 generic.go:356] "Generic (PLEG): container finished" podID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerID="23757736ace7eb3ffdae26db7a5ea5d4493c4f615e921c9e22e4405837b61c02" exitCode=0 Jul 22 02:32:12.295245 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:12.294862 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" event={"ID":"50a328b4-f01d-420c-a9dd-7185f30d1251","Type":"ContainerDied","Data":"23757736ace7eb3ffdae26db7a5ea5d4493c4f615e921c9e22e4405837b61c02"} Jul 22 02:32:15.312032 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:15.311996 2549 generic.go:356] "Generic (PLEG): container finished" podID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerID="fa32471498f3ca9b54aca2403483c0eb95b1d35f812fa945c9483c41638f6e5e" exitCode=0 Jul 22 02:32:15.312432 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:15.312055 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" event={"ID":"50a328b4-f01d-420c-a9dd-7185f30d1251","Type":"ContainerDied","Data":"fa32471498f3ca9b54aca2403483c0eb95b1d35f812fa945c9483c41638f6e5e"} Jul 22 02:32:23.346483 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:23.346444 2549 generic.go:356] "Generic (PLEG): container finished" podID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerID="9d7f234b03d6900420d7553a52e5a42fc132ac67ffe59c86306c5f7fd888c956" exitCode=0 Jul 22 02:32:23.346858 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:23.346501 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" event={"ID":"50a328b4-f01d-420c-a9dd-7185f30d1251","Type":"ContainerDied","Data":"9d7f234b03d6900420d7553a52e5a42fc132ac67ffe59c86306c5f7fd888c956"} Jul 22 02:32:24.473975 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.473952 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:24.503890 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.503864 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util\") pod \"50a328b4-f01d-420c-a9dd-7185f30d1251\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " Jul 22 02:32:24.504031 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.503931 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdpg6\" (UniqueName: \"kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6\") pod \"50a328b4-f01d-420c-a9dd-7185f30d1251\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " Jul 22 02:32:24.504031 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.503959 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle\") pod \"50a328b4-f01d-420c-a9dd-7185f30d1251\" (UID: \"50a328b4-f01d-420c-a9dd-7185f30d1251\") " Jul 22 02:32:24.504849 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.504816 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle" (OuterVolumeSpecName: "bundle") pod "50a328b4-f01d-420c-a9dd-7185f30d1251" (UID: "50a328b4-f01d-420c-a9dd-7185f30d1251"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:32:24.506126 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.506094 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6" (OuterVolumeSpecName: "kube-api-access-vdpg6") pod "50a328b4-f01d-420c-a9dd-7185f30d1251" (UID: "50a328b4-f01d-420c-a9dd-7185f30d1251"). InnerVolumeSpecName "kube-api-access-vdpg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:32:24.510361 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.510336 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util" (OuterVolumeSpecName: "util") pod "50a328b4-f01d-420c-a9dd-7185f30d1251" (UID: "50a328b4-f01d-420c-a9dd-7185f30d1251"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:32:24.604956 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.604881 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:24.604956 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.604908 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdpg6\" (UniqueName: \"kubernetes.io/projected/50a328b4-f01d-420c-a9dd-7185f30d1251-kube-api-access-vdpg6\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:24.604956 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:24.604918 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50a328b4-f01d-420c-a9dd-7185f30d1251-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:25.355046 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:25.355014 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" event={"ID":"50a328b4-f01d-420c-a9dd-7185f30d1251","Type":"ContainerDied","Data":"a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af"} Jul 22 02:32:25.355046 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:25.355049 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540snc42" Jul 22 02:32:25.355254 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:25.355059 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a14a18ca06fc6d0c3dbb1f36b8213f760252f2a96c056ed79233164b49f1e8af" Jul 22 02:32:32.233825 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.233792 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k"] Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234055 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="pull" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234066 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="pull" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234087 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="util" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234092 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="util" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234100 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="extract" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234106 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="extract" Jul 22 02:32:32.234205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.234154 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a328b4-f01d-420c-a9dd-7185f30d1251" containerName="extract" Jul 22 02:32:32.238025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.237999 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.243509 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.243456 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-55gqt\"" Jul 22 02:32:32.243773 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.243757 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 02:32:32.243847 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.243774 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:32:32.262955 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.262927 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k"] Jul 22 02:32:32.268180 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.268154 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af8ef02-266c-4e1f-8280-e86e9f6adf25-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.268289 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.268214 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzr6\" (UniqueName: \"kubernetes.io/projected/9af8ef02-266c-4e1f-8280-e86e9f6adf25-kube-api-access-5wzr6\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.368872 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.368833 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af8ef02-266c-4e1f-8280-e86e9f6adf25-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.369046 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.368886 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzr6\" (UniqueName: \"kubernetes.io/projected/9af8ef02-266c-4e1f-8280-e86e9f6adf25-kube-api-access-5wzr6\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.369302 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.369275 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9af8ef02-266c-4e1f-8280-e86e9f6adf25-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.397093 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.397067 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzr6\" (UniqueName: \"kubernetes.io/projected/9af8ef02-266c-4e1f-8280-e86e9f6adf25-kube-api-access-5wzr6\") pod \"cert-manager-operator-controller-manager-55c444bf86-9h97k\" (UID: \"9af8ef02-266c-4e1f-8280-e86e9f6adf25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.547750 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.547660 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" Jul 22 02:32:32.728205 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:32.728183 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k"] Jul 22 02:32:32.730462 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:32.730435 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af8ef02_266c_4e1f_8280_e86e9f6adf25.slice/crio-c3679f937de53bdbc2deef4f849a92800245e48a2010b81768a472844a0c1f15 WatchSource:0}: Error finding container c3679f937de53bdbc2deef4f849a92800245e48a2010b81768a472844a0c1f15: Status 404 returned error can't find the container with id c3679f937de53bdbc2deef4f849a92800245e48a2010b81768a472844a0c1f15 Jul 22 02:32:33.384870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:33.384830 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" event={"ID":"9af8ef02-266c-4e1f-8280-e86e9f6adf25","Type":"ContainerStarted","Data":"c3679f937de53bdbc2deef4f849a92800245e48a2010b81768a472844a0c1f15"} Jul 22 02:32:41.418018 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:41.417935 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" event={"ID":"9af8ef02-266c-4e1f-8280-e86e9f6adf25","Type":"ContainerStarted","Data":"97e7a3f44e4465f65feef9de731331024975a0cf0cf83d4d2fdf01580ec9aa9d"} Jul 22 02:32:41.441287 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:41.441238 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9h97k" podStartSLOduration=1.081541964 podStartE2EDuration="9.441223328s" totalImagesPullingTime="8.359681364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:32:41.439226731 +0000 UTC m=+252.596239018" watchObservedRunningTime="2026-07-22 02:32:41.441223328 +0000 UTC m=+252.598235611" Jul 22 02:32:42.233636 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.233601 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zqbd"] Jul 22 02:32:42.236074 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.236055 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.239220 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.239191 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-txm9b\"" Jul 22 02:32:42.239964 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.239948 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 02:32:42.240130 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.240112 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 02:32:42.265984 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.265940 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zqbd"] Jul 22 02:32:42.347827 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.347787 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqm6\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-kube-api-access-9xqm6\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.347827 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.347829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.449273 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.449239 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqm6\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-kube-api-access-9xqm6\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.449273 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.449277 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.457158 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.457133 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.457980 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.457963 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqm6\" (UniqueName: \"kubernetes.io/projected/19798bf9-12aa-455f-93ba-a77daa47dca0-kube-api-access-9xqm6\") pod \"cert-manager-webhook-6486ffd997-5zqbd\" (UID: \"19798bf9-12aa-455f-93ba-a77daa47dca0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.555200 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.555118 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:42.682383 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:42.682350 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5zqbd"] Jul 22 02:32:42.684869 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:42.684840 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19798bf9_12aa_455f_93ba_a77daa47dca0.slice/crio-1d49adc23bb979f0cf9369c9627f07c80a61ffd75cfedf267818f924fe74f6e6 WatchSource:0}: Error finding container 1d49adc23bb979f0cf9369c9627f07c80a61ffd75cfedf267818f924fe74f6e6: Status 404 returned error can't find the container with id 1d49adc23bb979f0cf9369c9627f07c80a61ffd75cfedf267818f924fe74f6e6 Jul 22 02:32:43.427139 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.427102 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" event={"ID":"19798bf9-12aa-455f-93ba-a77daa47dca0","Type":"ContainerStarted","Data":"1d49adc23bb979f0cf9369c9627f07c80a61ffd75cfedf267818f924fe74f6e6"} Jul 22 02:32:43.447087 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.447055 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw"] Jul 22 02:32:43.449650 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.449623 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.452158 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.452136 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 02:32:43.452295 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.452144 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 02:32:43.452991 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.452973 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fblmd\"" Jul 22 02:32:43.460092 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.460038 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw"] Jul 22 02:32:43.558366 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.558332 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrgl\" (UniqueName: \"kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.558543 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.558371 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.558543 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.558430 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.659593 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.659554 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrgl\" (UniqueName: \"kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.659593 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.659599 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.659799 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.659624 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.659945 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.659929 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.660020 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.660003 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.668054 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.668030 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrgl\" (UniqueName: \"kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.762563 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.762481 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:43.900877 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:43.900851 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw"] Jul 22 02:32:43.903536 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:43.903507 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc428ac0_b889_45c8_9b8b_2e9a47b46bfb.slice/crio-fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370 WatchSource:0}: Error finding container fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370: Status 404 returned error can't find the container with id fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370 Jul 22 02:32:44.433229 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:44.433196 2549 generic.go:356] "Generic (PLEG): container finished" podID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerID="541a17f7d8cfb54e7a0f70b2625f3ce598cc8ebfef9bd863fa3cf6d9007c0b52" exitCode=0 Jul 22 02:32:44.433437 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:44.433282 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" event={"ID":"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb","Type":"ContainerDied","Data":"541a17f7d8cfb54e7a0f70b2625f3ce598cc8ebfef9bd863fa3cf6d9007c0b52"} Jul 22 02:32:44.433437 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:44.433340 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" event={"ID":"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb","Type":"ContainerStarted","Data":"fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370"} Jul 22 02:32:45.321142 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.321109 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc"] Jul 22 02:32:45.323865 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.323848 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.326379 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.326362 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dhpbx\"" Jul 22 02:32:45.334504 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.334475 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc"] Jul 22 02:32:45.376376 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.376344 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqc6f\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-kube-api-access-wqc6f\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.376615 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.376383 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.477585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.477534 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqc6f\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-kube-api-access-wqc6f\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.477585 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.477591 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.486566 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.486528 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.486754 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.486729 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqc6f\" (UniqueName: \"kubernetes.io/projected/5108d6a7-09cd-46d7-a419-feb0ef2f7409-kube-api-access-wqc6f\") pod \"cert-manager-cainjector-559cdd8c87-2l7fc\" (UID: \"5108d6a7-09cd-46d7-a419-feb0ef2f7409\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.633625 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.633545 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" Jul 22 02:32:45.785471 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:45.785437 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc"] Jul 22 02:32:45.787354 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:45.787319 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5108d6a7_09cd_46d7_a419_feb0ef2f7409.slice/crio-1ecac2e7f0d5ca918ab6513fd0f5d0b46846ec51833fc6fe939d7b46a68518ac WatchSource:0}: Error finding container 1ecac2e7f0d5ca918ab6513fd0f5d0b46846ec51833fc6fe939d7b46a68518ac: Status 404 returned error can't find the container with id 1ecac2e7f0d5ca918ab6513fd0f5d0b46846ec51833fc6fe939d7b46a68518ac Jul 22 02:32:46.444297 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:46.444262 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" event={"ID":"5108d6a7-09cd-46d7-a419-feb0ef2f7409","Type":"ContainerStarted","Data":"1ecac2e7f0d5ca918ab6513fd0f5d0b46846ec51833fc6fe939d7b46a68518ac"} Jul 22 02:32:48.453863 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.453832 2549 generic.go:356] "Generic (PLEG): container finished" podID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerID="2512ecc59cb1962c653c2688aee4dcd21fe75c94618d70a75b2c1207e8f4ac8c" exitCode=0 Jul 22 02:32:48.454317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.453922 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" event={"ID":"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb","Type":"ContainerDied","Data":"2512ecc59cb1962c653c2688aee4dcd21fe75c94618d70a75b2c1207e8f4ac8c"} Jul 22 02:32:48.458131 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.458104 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" event={"ID":"5108d6a7-09cd-46d7-a419-feb0ef2f7409","Type":"ContainerStarted","Data":"245d058f43c6dfd045bffd72c591e91943f83e4331ca95bcd601508e5a829106"} Jul 22 02:32:48.459647 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.459627 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" event={"ID":"19798bf9-12aa-455f-93ba-a77daa47dca0","Type":"ContainerStarted","Data":"294b4c8692996237604f580f3b08818c90bdc607280bca97234a499351d03731"} Jul 22 02:32:48.460125 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.460106 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:48.510881 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.510825 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2l7fc" podStartSLOduration=1.833714686 podStartE2EDuration="3.510805488s" totalImagesPullingTime="1.677090802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:32:48.489591684 +0000 UTC m=+259.646603978" watchObservedRunningTime="2026-07-22 02:32:48.510805488 +0000 UTC m=+259.667817773" Jul 22 02:32:48.511088 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:48.510936 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" podStartSLOduration=1.764493216 podStartE2EDuration="6.510926013s" totalImagesPullingTime="4.746432797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:32:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:32:48.509542283 +0000 UTC m=+259.666554565" watchObservedRunningTime="2026-07-22 02:32:48.510926013 +0000 UTC m=+259.667938298" Jul 22 02:32:49.465172 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:49.465098 2549 generic.go:356] "Generic (PLEG): container finished" podID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerID="bdbc74104e023cabfc180828d489ff5d075a806e21d6041893f32818051d55e3" exitCode=0 Jul 22 02:32:49.465556 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:49.465190 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" event={"ID":"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb","Type":"ContainerDied","Data":"bdbc74104e023cabfc180828d489ff5d075a806e21d6041893f32818051d55e3"} Jul 22 02:32:50.591318 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.591061 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:50.729464 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.729347 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle\") pod \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " Jul 22 02:32:50.729624 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.729495 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util\") pod \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " Jul 22 02:32:50.729624 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.729527 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltrgl\" (UniqueName: \"kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl\") pod \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\" (UID: \"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb\") " Jul 22 02:32:50.729835 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.729811 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle" (OuterVolumeSpecName: "bundle") pod "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" (UID: "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:32:50.731701 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.731680 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl" (OuterVolumeSpecName: "kube-api-access-ltrgl") pod "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" (UID: "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb"). InnerVolumeSpecName "kube-api-access-ltrgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:32:50.733949 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.733909 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util" (OuterVolumeSpecName: "util") pod "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" (UID: "fc428ac0-b889-45c8-9b8b-2e9a47b46bfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:32:50.830732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.830684 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:50.830732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.830725 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrgl\" (UniqueName: \"kubernetes.io/projected/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-kube-api-access-ltrgl\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:50.830732 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:50.830738 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc428ac0-b889-45c8-9b8b-2e9a47b46bfb-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:32:51.474826 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:51.474788 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" event={"ID":"fc428ac0-b889-45c8-9b8b-2e9a47b46bfb","Type":"ContainerDied","Data":"fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370"} Jul 22 02:32:51.474826 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:51.474812 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcr4bw" Jul 22 02:32:51.475026 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:51.474841 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa983ddf5ac46cfc326f6ac8e993cbd6aba7a885556813d521c3e89831632370" Jul 22 02:32:54.467711 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:54.467682 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5zqbd" Jul 22 02:32:56.934973 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.934935 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6"] Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935238 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="pull" Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935249 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="pull" Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935272 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="util" Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935278 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="util" Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935289 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="extract" Jul 22 02:32:56.935337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935294 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="extract" Jul 22 02:32:56.935549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.935350 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc428ac0-b889-45c8-9b8b-2e9a47b46bfb" containerName="extract" Jul 22 02:32:56.955971 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.955939 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6"] Jul 22 02:32:56.956120 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.956033 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:56.958716 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.958694 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 02:32:56.959893 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.959872 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 02:32:56.960017 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:56.959878 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-sk8bx\"" Jul 22 02:32:57.084239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.084211 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8baee0-c70e-40ae-8c83-ad5937999806-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.084439 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.084285 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq4dg\" (UniqueName: \"kubernetes.io/projected/7d8baee0-c70e-40ae-8c83-ad5937999806-kube-api-access-cq4dg\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.185407 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.185330 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq4dg\" (UniqueName: \"kubernetes.io/projected/7d8baee0-c70e-40ae-8c83-ad5937999806-kube-api-access-cq4dg\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.185407 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.185372 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8baee0-c70e-40ae-8c83-ad5937999806-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.185770 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.185754 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d8baee0-c70e-40ae-8c83-ad5937999806-tmp\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.194739 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.194713 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq4dg\" (UniqueName: \"kubernetes.io/projected/7d8baee0-c70e-40ae-8c83-ad5937999806-kube-api-access-cq4dg\") pod \"openshift-lws-operator-fd8ccff4c-f4tc6\" (UID: \"7d8baee0-c70e-40ae-8c83-ad5937999806\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.265433 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.265384 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" Jul 22 02:32:57.420648 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.420616 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6"] Jul 22 02:32:57.423746 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:32:57.423721 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d8baee0_c70e_40ae_8c83_ad5937999806.slice/crio-149dffdcfcd66db18be137c26448952a79f77855fec1f9e07fc2bda05fc9dec2 WatchSource:0}: Error finding container 149dffdcfcd66db18be137c26448952a79f77855fec1f9e07fc2bda05fc9dec2: Status 404 returned error can't find the container with id 149dffdcfcd66db18be137c26448952a79f77855fec1f9e07fc2bda05fc9dec2 Jul 22 02:32:57.501701 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:32:57.501621 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" event={"ID":"7d8baee0-c70e-40ae-8c83-ad5937999806","Type":"ContainerStarted","Data":"149dffdcfcd66db18be137c26448952a79f77855fec1f9e07fc2bda05fc9dec2"} Jul 22 02:33:00.518191 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:00.518156 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" event={"ID":"7d8baee0-c70e-40ae-8c83-ad5937999806","Type":"ContainerStarted","Data":"8517dde8336dd5fbe37604cc0233296f49131487380962790300e22b6bf195b8"} Jul 22 02:33:00.563349 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:00.563289 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-f4tc6" podStartSLOduration=1.847991139 podStartE2EDuration="4.563270621s" totalImagesPullingTime="2.715279482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:00.561803132 +0000 UTC m=+271.718815415" watchObservedRunningTime="2026-07-22 02:33:00.563270621 +0000 UTC m=+271.720282904" Jul 22 02:33:02.989994 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:02.989954 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5"] Jul 22 02:33:02.992710 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:02.992682 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:02.995311 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:02.995285 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 02:33:02.995450 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:02.995310 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 02:33:02.996225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:02.996206 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fblmd\"" Jul 22 02:33:03.003393 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.003367 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5"] Jul 22 02:33:03.136907 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.136863 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6tr\" (UniqueName: \"kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.137081 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.136950 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.137081 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.136976 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.238015 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.237961 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.238199 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.238050 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6tr\" (UniqueName: \"kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.238199 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.238104 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.238365 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.238346 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.238480 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.238460 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.247190 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.247108 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6tr\" (UniqueName: \"kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.301794 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.301751 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:03.464720 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.464687 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5"] Jul 22 02:33:03.467329 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:33:03.467301 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e059fda_2de0_44fa_8c53_671e1db287f4.slice/crio-3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba WatchSource:0}: Error finding container 3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba: Status 404 returned error can't find the container with id 3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba Jul 22 02:33:03.532971 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.532947 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerStarted","Data":"f769b2156be82891cd704b6bb0c9a6c68bb12a6348f42f0715bbf4852882cb22"} Jul 22 02:33:03.533078 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:03.532997 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerStarted","Data":"3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba"} Jul 22 02:33:04.538225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:04.538145 2549 generic.go:356] "Generic (PLEG): container finished" podID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerID="f769b2156be82891cd704b6bb0c9a6c68bb12a6348f42f0715bbf4852882cb22" exitCode=0 Jul 22 02:33:04.538225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:04.538218 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerDied","Data":"f769b2156be82891cd704b6bb0c9a6c68bb12a6348f42f0715bbf4852882cb22"} Jul 22 02:33:05.544797 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:05.544766 2549 generic.go:356] "Generic (PLEG): container finished" podID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerID="91d33feef91f55967c44c74fb16775ba076423346b2b381632930ec33dbd44aa" exitCode=0 Jul 22 02:33:05.545209 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:05.544862 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerDied","Data":"91d33feef91f55967c44c74fb16775ba076423346b2b381632930ec33dbd44aa"} Jul 22 02:33:06.552074 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:06.552039 2549 generic.go:356] "Generic (PLEG): container finished" podID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerID="75e5f2bf95d31f0c599674c5e5e2d741c7a7bedc870708050f8ca2e4697c6ee7" exitCode=0 Jul 22 02:33:06.552493 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:06.552124 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerDied","Data":"75e5f2bf95d31f0c599674c5e5e2d741c7a7bedc870708050f8ca2e4697c6ee7"} Jul 22 02:33:07.684806 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.684782 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:07.779823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.779792 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util\") pod \"1e059fda-2de0-44fa-8c53-671e1db287f4\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " Jul 22 02:33:07.779823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.779826 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle\") pod \"1e059fda-2de0-44fa-8c53-671e1db287f4\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " Jul 22 02:33:07.780072 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.779896 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6tr\" (UniqueName: \"kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr\") pod \"1e059fda-2de0-44fa-8c53-671e1db287f4\" (UID: \"1e059fda-2de0-44fa-8c53-671e1db287f4\") " Jul 22 02:33:07.780571 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.780546 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle" (OuterVolumeSpecName: "bundle") pod "1e059fda-2de0-44fa-8c53-671e1db287f4" (UID: "1e059fda-2de0-44fa-8c53-671e1db287f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:33:07.782049 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.782019 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr" (OuterVolumeSpecName: "kube-api-access-6q6tr") pod "1e059fda-2de0-44fa-8c53-671e1db287f4" (UID: "1e059fda-2de0-44fa-8c53-671e1db287f4"). InnerVolumeSpecName "kube-api-access-6q6tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:33:07.785057 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.785036 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util" (OuterVolumeSpecName: "util") pod "1e059fda-2de0-44fa-8c53-671e1db287f4" (UID: "1e059fda-2de0-44fa-8c53-671e1db287f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:33:07.880908 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.880819 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6tr\" (UniqueName: \"kubernetes.io/projected/1e059fda-2de0-44fa-8c53-671e1db287f4-kube-api-access-6q6tr\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:33:07.880908 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.880849 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:33:07.880908 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:07.880863 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1e059fda-2de0-44fa-8c53-671e1db287f4-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:33:08.562620 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:08.562586 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" event={"ID":"1e059fda-2de0-44fa-8c53-671e1db287f4","Type":"ContainerDied","Data":"3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba"} Jul 22 02:33:08.562788 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:08.562642 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc4ea04f99330218cdda623777cba8978697f295ac268fe5c624e3e34c4d8ba" Jul 22 02:33:08.562788 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:08.562680 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afqctg5" Jul 22 02:33:16.282129 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282099 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf"] Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282378 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="pull" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282388 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="pull" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282403 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="util" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282408 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="util" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282435 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="extract" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282442 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="extract" Jul 22 02:33:16.282651 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.282508 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e059fda-2de0-44fa-8c53-671e1db287f4" containerName="extract" Jul 22 02:33:16.287762 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.287746 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.295691 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.295668 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gwnxn\"" Jul 22 02:33:16.296011 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.295996 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 02:33:16.296190 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.296175 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 02:33:16.296238 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.296180 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 02:33:16.300138 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.300121 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 02:33:16.308764 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.308739 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf"] Jul 22 02:33:16.343220 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.343187 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.343329 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.343225 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.343329 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.343243 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6dv\" (UniqueName: \"kubernetes.io/projected/65435ad2-7f85-4955-ac4f-b2954847459e-kube-api-access-rv6dv\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.343329 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.343298 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.343329 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.343325 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444052 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444003 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444424 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444382 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444507 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444459 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6dv\" (UniqueName: \"kubernetes.io/projected/65435ad2-7f85-4955-ac4f-b2954847459e-kube-api-access-rv6dv\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444507 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444468 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444587 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444568 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.444748 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.444722 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.446906 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.446881 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65435ad2-7f85-4955-ac4f-b2954847459e-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.447139 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.447122 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.447186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.447141 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65435ad2-7f85-4955-ac4f-b2954847459e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.456312 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.456288 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6dv\" (UniqueName: \"kubernetes.io/projected/65435ad2-7f85-4955-ac4f-b2954847459e-kube-api-access-rv6dv\") pod \"opendatahub-operator-controller-manager-86c79b8765-phdrf\" (UID: \"65435ad2-7f85-4955-ac4f-b2954847459e\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.603436 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.603338 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:16.741722 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:16.741698 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf"] Jul 22 02:33:16.744490 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:33:16.744464 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65435ad2_7f85_4955_ac4f_b2954847459e.slice/crio-0485624d1fdd686a0cfd7ef3bfdfed55dabad7e11272c9453f834bc1ac181be9 WatchSource:0}: Error finding container 0485624d1fdd686a0cfd7ef3bfdfed55dabad7e11272c9453f834bc1ac181be9: Status 404 returned error can't find the container with id 0485624d1fdd686a0cfd7ef3bfdfed55dabad7e11272c9453f834bc1ac181be9 Jul 22 02:33:17.605602 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:17.605562 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" event={"ID":"65435ad2-7f85-4955-ac4f-b2954847459e","Type":"ContainerStarted","Data":"0485624d1fdd686a0cfd7ef3bfdfed55dabad7e11272c9453f834bc1ac181be9"} Jul 22 02:33:20.622139 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:20.622103 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" event={"ID":"65435ad2-7f85-4955-ac4f-b2954847459e","Type":"ContainerStarted","Data":"deb6458be3f90c759b55a6c3854ec48fe4baba03048e2cd4e30bdd3fb54e5851"} Jul 22 02:33:21.630165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:21.630085 2549 generic.go:356] "Generic (PLEG): container finished" podID="65435ad2-7f85-4955-ac4f-b2954847459e" containerID="deb6458be3f90c759b55a6c3854ec48fe4baba03048e2cd4e30bdd3fb54e5851" exitCode=0 Jul 22 02:33:21.630165 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:21.630125 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" event={"ID":"65435ad2-7f85-4955-ac4f-b2954847459e","Type":"ContainerDied","Data":"deb6458be3f90c759b55a6c3854ec48fe4baba03048e2cd4e30bdd3fb54e5851"} Jul 22 02:33:22.636148 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:22.636115 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" event={"ID":"65435ad2-7f85-4955-ac4f-b2954847459e","Type":"ContainerStarted","Data":"0a86f002b8cb30671da2e3cccafc680d6f3d8955fc2597ce7332af33d49b7aeb"} Jul 22 02:33:22.636555 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:22.636240 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:22.662151 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:22.662102 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" podStartSLOduration=1.900986982 podStartE2EDuration="6.662083524s" totalImagesPullingTime="3.728237199s" totalInitContainerRuntime="1.032859343s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:22.66083363 +0000 UTC m=+293.817845904" watchObservedRunningTime="2026-07-22 02:33:22.662083524 +0000 UTC m=+293.819095807" Jul 22 02:33:29.247346 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:29.247315 2549 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 02:33:33.643030 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.642997 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-phdrf" Jul 22 02:33:33.987609 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.987567 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n"] Jul 22 02:33:33.991638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.991616 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:33.996996 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.996977 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nlphm\"" Jul 22 02:33:33.997121 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.997039 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 02:33:33.997843 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.997741 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 02:33:33.998011 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:33.997993 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 02:33:34.004158 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.004136 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n"] Jul 22 02:33:34.098080 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.098042 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.098266 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.098104 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmchk\" (UniqueName: \"kubernetes.io/projected/714e8fc9-0bc7-415b-8120-e1cd890667ac-kube-api-access-rmchk\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.098266 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.098204 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.098374 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.098272 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/714e8fc9-0bc7-415b-8120-e1cd890667ac-manager-config\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.199762 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.199712 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.199762 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.199772 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmchk\" (UniqueName: \"kubernetes.io/projected/714e8fc9-0bc7-415b-8120-e1cd890667ac-kube-api-access-rmchk\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.200020 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.199798 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.200020 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.199835 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/714e8fc9-0bc7-415b-8120-e1cd890667ac-manager-config\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.200572 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.200471 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/714e8fc9-0bc7-415b-8120-e1cd890667ac-manager-config\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.202364 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.202337 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.202471 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.202375 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/714e8fc9-0bc7-415b-8120-e1cd890667ac-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.210793 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.210772 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmchk\" (UniqueName: \"kubernetes.io/projected/714e8fc9-0bc7-415b-8120-e1cd890667ac-kube-api-access-rmchk\") pod \"lws-controller-manager-57d76bdcd7-hnf5n\" (UID: \"714e8fc9-0bc7-415b-8120-e1cd890667ac\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.300491 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.300392 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:34.435365 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.435333 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n"] Jul 22 02:33:34.438218 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:33:34.438194 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714e8fc9_0bc7_415b_8120_e1cd890667ac.slice/crio-ab8ad399dbf4b62bb9fee6ae550678b8fe40b574342d8a72037bbdedd25d52e9 WatchSource:0}: Error finding container ab8ad399dbf4b62bb9fee6ae550678b8fe40b574342d8a72037bbdedd25d52e9: Status 404 returned error can't find the container with id ab8ad399dbf4b62bb9fee6ae550678b8fe40b574342d8a72037bbdedd25d52e9 Jul 22 02:33:34.440108 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.440082 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:33:34.694349 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:34.694316 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" event={"ID":"714e8fc9-0bc7-415b-8120-e1cd890667ac","Type":"ContainerStarted","Data":"ab8ad399dbf4b62bb9fee6ae550678b8fe40b574342d8a72037bbdedd25d52e9"} Jul 22 02:33:36.705768 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:36.705734 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" event={"ID":"714e8fc9-0bc7-415b-8120-e1cd890667ac","Type":"ContainerStarted","Data":"fa810975bab0cd45c596173c1f343205439fe67faa3be11a823dc85d1774dffe"} Jul 22 02:33:36.706186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:36.705845 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:36.744672 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:36.744623 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" podStartSLOduration=2.11157297 podStartE2EDuration="3.744609279s" totalImagesPullingTime="1.633036309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:36.74360463 +0000 UTC m=+307.900616910" watchObservedRunningTime="2026-07-22 02:33:36.744609279 +0000 UTC m=+307.901621562" Jul 22 02:33:45.579061 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.579027 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt"] Jul 22 02:33:45.583334 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.583316 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.586394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.586375 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gpks2\"" Jul 22 02:33:45.586949 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.586931 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 02:33:45.587024 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.586935 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 02:33:45.587120 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.586939 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 02:33:45.625164 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.625133 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt"] Jul 22 02:33:45.703145 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703110 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703372 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703337 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703380 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32838d19-072b-4b8b-86cd-ff53120ed979-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703440 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703492 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9dv\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-kube-api-access-5r9dv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703544 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703816 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703612 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703816 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703681 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.703816 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.703712 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32838d19-072b-4b8b-86cd-ff53120ed979-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.804992 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.804954 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.804992 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.804992 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32838d19-072b-4b8b-86cd-ff53120ed979-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805014 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805042 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9dv\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-kube-api-access-5r9dv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805066 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805106 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805135 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805159 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32838d19-072b-4b8b-86cd-ff53120ed979-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805242 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805205 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805623 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805491 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805623 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805514 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.805779 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.806069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.806069 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.805940 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32838d19-072b-4b8b-86cd-ff53120ed979-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.807558 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.807539 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32838d19-072b-4b8b-86cd-ff53120ed979-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.807703 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.807687 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32838d19-072b-4b8b-86cd-ff53120ed979-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.822130 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.822100 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.823344 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.823320 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9dv\" (UniqueName: \"kubernetes.io/projected/32838d19-072b-4b8b-86cd-ff53120ed979-kube-api-access-5r9dv\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j8wkt\" (UID: \"32838d19-072b-4b8b-86cd-ff53120ed979\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:45.893638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:45.893565 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:46.042279 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.042252 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt"] Jul 22 02:33:46.044080 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:33:46.044053 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32838d19_072b_4b8b_86cd_ff53120ed979.slice/crio-02025bd66877cb08851baea3ef50b08ad3aa68b6b9ab1abad556414611311965 WatchSource:0}: Error finding container 02025bd66877cb08851baea3ef50b08ad3aa68b6b9ab1abad556414611311965: Status 404 returned error can't find the container with id 02025bd66877cb08851baea3ef50b08ad3aa68b6b9ab1abad556414611311965 Jul 22 02:33:46.243944 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.243902 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx"] Jul 22 02:33:46.248323 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.248300 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.253638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.253601 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 02:33:46.253638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.253607 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 02:33:46.253638 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.253608 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 02:33:46.253987 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.253955 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-68hhv\"" Jul 22 02:33:46.271032 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.270999 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx"] Jul 22 02:33:46.410787 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410759 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3ff7db-5356-4305-ab03-1f6892c1a38b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411006 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410799 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411006 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410818 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411006 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410872 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7t4l\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-kube-api-access-f7t4l\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411006 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410911 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411006 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.410974 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.411198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.411013 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.511927 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.511835 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3ff7db-5356-4305-ab03-1f6892c1a38b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.511927 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.511891 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.512190 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.512021 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.512190 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.512091 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7t4l\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-kube-api-access-f7t4l\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.512190 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.512133 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.512337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.512213 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.512337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.512240 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.513100 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.513075 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.514437 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.514399 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf3ff7db-5356-4305-ab03-1f6892c1a38b-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.514582 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.514565 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.514892 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.514875 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.514972 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.514950 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.521530 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.521505 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7t4l\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-kube-api-access-f7t4l\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.522191 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.522171 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf3ff7db-5356-4305-ab03-1f6892c1a38b-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-7kqgx\" (UID: \"bf3ff7db-5356-4305-ab03-1f6892c1a38b\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.557322 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.557294 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:46.696465 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.696431 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx"] Jul 22 02:33:46.753721 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.753688 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" event={"ID":"bf3ff7db-5356-4305-ab03-1f6892c1a38b","Type":"ContainerStarted","Data":"8486258684b7173d81cb481f3a1520c88f03ac7c34c30d78725fbbc6803ae95f"} Jul 22 02:33:46.754958 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:46.754926 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" event={"ID":"32838d19-072b-4b8b-86cd-ff53120ed979","Type":"ContainerStarted","Data":"02025bd66877cb08851baea3ef50b08ad3aa68b6b9ab1abad556414611311965"} Jul 22 02:33:47.715097 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:47.715061 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-hnf5n" Jul 22 02:33:50.143850 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.143752 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:33:50.144154 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.143851 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:33:50.144154 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.143892 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:33:50.145677 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.145647 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:33:50.145784 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.145704 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:33:50.777958 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.777915 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" event={"ID":"bf3ff7db-5356-4305-ab03-1f6892c1a38b","Type":"ContainerStarted","Data":"31fc7a32796c1a97aaceff48e9b50f5e0a8c976c6bb17f92f93ae1a5c3add722"} Jul 22 02:33:50.778167 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.777985 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:50.779569 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.779542 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" event={"ID":"32838d19-072b-4b8b-86cd-ff53120ed979","Type":"ContainerStarted","Data":"598765474a5fa5ce18ebcba48d2917d0e62d8535fa4dd64b5fced67c823ccaa4"} Jul 22 02:33:50.828903 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.828834 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" podStartSLOduration=1.384951187 podStartE2EDuration="4.828818406s" totalImagesPullingTime="3.443867219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:50.821864415 +0000 UTC m=+321.978876697" watchObservedRunningTime="2026-07-22 02:33:50.828818406 +0000 UTC m=+321.985830689" Jul 22 02:33:50.856348 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.856284 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" podStartSLOduration=1.758874456 podStartE2EDuration="5.856262959s" totalImagesPullingTime="4.097388503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:50.854106877 +0000 UTC m=+322.011119160" watchObservedRunningTime="2026-07-22 02:33:50.856262959 +0000 UTC m=+322.013275243" Jul 22 02:33:50.894136 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:50.894101 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:51.785853 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:51.785826 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-7kqgx" Jul 22 02:33:51.899607 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:51.899567 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:52.789072 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:52.789037 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:33:52.789940 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:33:52.789923 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j8wkt" Jul 22 02:34:52.990579 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:52.990535 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd"] Jul 22 02:34:52.993657 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:52.993633 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:52.996328 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:52.996302 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 02:34:52.996461 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:52.996308 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-srhxs\"" Jul 22 02:34:52.997101 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:52.997082 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 02:34:53.002532 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.002510 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd"] Jul 22 02:34:53.109862 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.109829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.110034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.109876 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.110034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.109915 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrg5f\" (UniqueName: \"kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.210945 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.210914 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.211090 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.210956 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrg5f\" (UniqueName: \"kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.211090 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.211017 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.211360 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.211336 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.211451 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.211380 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.219937 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.219916 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrg5f\" (UniqueName: \"kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.302965 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.302881 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:53.430221 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.430197 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd"] Jul 22 02:34:53.777763 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.777732 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f"] Jul 22 02:34:53.780567 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.780551 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.789381 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.789360 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f"] Jul 22 02:34:53.816343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.816321 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.816489 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.816349 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.816489 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.816382 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htprd\" (UniqueName: \"kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.916942 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.916914 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.917112 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.916948 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.917112 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.917076 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htprd\" (UniqueName: \"kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.917321 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.917301 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.917394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.917303 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:53.925529 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:53.925511 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htprd\" (UniqueName: \"kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:54.061383 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.061298 2549 generic.go:356] "Generic (PLEG): container finished" podID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerID="2fe0d4b63b6cc96eeb2730dabe80c4fb2e15c2674ddf8bd0a5bf2bd30f584471" exitCode=0 Jul 22 02:34:54.061383 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.061352 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" event={"ID":"456c911f-b427-4e93-a6c8-09bdbbdce24b","Type":"ContainerDied","Data":"2fe0d4b63b6cc96eeb2730dabe80c4fb2e15c2674ddf8bd0a5bf2bd30f584471"} Jul 22 02:34:54.061793 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.061386 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" event={"ID":"456c911f-b427-4e93-a6c8-09bdbbdce24b","Type":"ContainerStarted","Data":"9bf03ae6ef2ca14c1f0f62e8d543b85f5385562798da7a091405026514cf6300"} Jul 22 02:34:54.090217 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.090193 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:54.220652 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.220629 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f"] Jul 22 02:34:54.222602 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:34:54.222575 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4a216d_c4a3_499d_b410_b055eef44fc8.slice/crio-dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0 WatchSource:0}: Error finding container dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0: Status 404 returned error can't find the container with id dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0 Jul 22 02:34:54.420108 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.420072 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz"] Jul 22 02:34:54.422728 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.422710 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.445811 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.445788 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz"] Jul 22 02:34:54.522085 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.522054 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.522085 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.522087 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2jp\" (UniqueName: \"kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.522297 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.522175 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.623187 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.623108 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.623187 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.623144 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2jp\" (UniqueName: \"kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.623358 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.623187 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.623515 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.623494 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.623558 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.623524 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.636788 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.636764 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2jp\" (UniqueName: \"kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.731285 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.731249 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:54.857582 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.857554 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t"] Jul 22 02:34:54.860234 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.860211 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:54.897265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.897200 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t"] Jul 22 02:34:54.899578 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.899554 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz"] Jul 22 02:34:54.926204 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.926180 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:54.926308 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.926227 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6ht\" (UniqueName: \"kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:54.926370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:54.926310 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:54.936314 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:34:54.936281 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc4d111_f49f_47b7_9c02_94523f4f880b.slice/crio-8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f WatchSource:0}: Error finding container 8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f: Status 404 returned error can't find the container with id 8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f Jul 22 02:34:55.026907 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.026874 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6ht\" (UniqueName: \"kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.027026 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.026925 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.027096 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.027024 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.027313 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.027290 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.027387 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.027321 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.040243 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.040224 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6ht\" (UniqueName: \"kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.068138 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.068105 2549 generic.go:356] "Generic (PLEG): container finished" podID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerID="c9eaeb1d88bcb1dfa2f85585e0a23e2717cbb77330d4837693d86536214c12e7" exitCode=0 Jul 22 02:34:55.068570 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.068190 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" event={"ID":"acc4d111-f49f-47b7-9c02-94523f4f880b","Type":"ContainerDied","Data":"c9eaeb1d88bcb1dfa2f85585e0a23e2717cbb77330d4837693d86536214c12e7"} Jul 22 02:34:55.068570 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.068235 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" event={"ID":"acc4d111-f49f-47b7-9c02-94523f4f880b","Type":"ContainerStarted","Data":"8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f"} Jul 22 02:34:55.070225 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.070204 2549 generic.go:356] "Generic (PLEG): container finished" podID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerID="b50c9979349525df4e0036967197275aa1eddd9139b81b3195ef99aabd94c4ef" exitCode=0 Jul 22 02:34:55.070319 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.070267 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" event={"ID":"456c911f-b427-4e93-a6c8-09bdbbdce24b","Type":"ContainerDied","Data":"b50c9979349525df4e0036967197275aa1eddd9139b81b3195ef99aabd94c4ef"} Jul 22 02:34:55.071931 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.071910 2549 generic.go:356] "Generic (PLEG): container finished" podID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerID="596da6dd5914c446d167e289fadc3f5802c6593b5bd816fa595676d65b055528" exitCode=0 Jul 22 02:34:55.072005 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.071959 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" event={"ID":"1b4a216d-c4a3-499d-b410-b055eef44fc8","Type":"ContainerDied","Data":"596da6dd5914c446d167e289fadc3f5802c6593b5bd816fa595676d65b055528"} Jul 22 02:34:55.072005 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.071990 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" event={"ID":"1b4a216d-c4a3-499d-b410-b055eef44fc8","Type":"ContainerStarted","Data":"dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0"} Jul 22 02:34:55.170295 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.170271 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:55.309626 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:55.309592 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t"] Jul 22 02:34:55.312903 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:34:55.312869 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653d7521_ee37_4a47_b0ec_d8d52c53a971.slice/crio-9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af WatchSource:0}: Error finding container 9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af: Status 404 returned error can't find the container with id 9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af Jul 22 02:34:56.079526 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:56.079496 2549 generic.go:356] "Generic (PLEG): container finished" podID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerID="e022057a651c94cc060f39a97240204f19dc80d3087714d75305c752b6116d55" exitCode=0 Jul 22 02:34:56.079952 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:56.079584 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" event={"ID":"456c911f-b427-4e93-a6c8-09bdbbdce24b","Type":"ContainerDied","Data":"e022057a651c94cc060f39a97240204f19dc80d3087714d75305c752b6116d55"} Jul 22 02:34:56.080916 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:56.080898 2549 generic.go:356] "Generic (PLEG): container finished" podID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerID="0ea190f121a039ad64a37874a508445373273305c113bb3b68d7038eb334290d" exitCode=0 Jul 22 02:34:56.080998 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:56.080976 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" event={"ID":"653d7521-ee37-4a47-b0ec-d8d52c53a971","Type":"ContainerDied","Data":"0ea190f121a039ad64a37874a508445373273305c113bb3b68d7038eb334290d"} Jul 22 02:34:56.081036 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:56.081021 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" event={"ID":"653d7521-ee37-4a47-b0ec-d8d52c53a971","Type":"ContainerStarted","Data":"9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af"} Jul 22 02:34:57.088274 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.088234 2549 generic.go:356] "Generic (PLEG): container finished" podID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerID="5ab94748629267ddafbf53f6aebc9457fd371879be411f24b37e692b2372e241" exitCode=0 Jul 22 02:34:57.088717 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.088316 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" event={"ID":"acc4d111-f49f-47b7-9c02-94523f4f880b","Type":"ContainerDied","Data":"5ab94748629267ddafbf53f6aebc9457fd371879be411f24b37e692b2372e241"} Jul 22 02:34:57.090277 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.090255 2549 generic.go:356] "Generic (PLEG): container finished" podID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerID="fd90d3af92168e889356fca7615738a11b697cd8b3ab6d48f0f1870da1256a9d" exitCode=0 Jul 22 02:34:57.090368 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.090300 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" event={"ID":"1b4a216d-c4a3-499d-b410-b055eef44fc8","Type":"ContainerDied","Data":"fd90d3af92168e889356fca7615738a11b697cd8b3ab6d48f0f1870da1256a9d"} Jul 22 02:34:57.092351 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.092331 2549 generic.go:356] "Generic (PLEG): container finished" podID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerID="214b5006cdcd77265ec95da23261f869c455bd8a0c4f43412c18af246c91a395" exitCode=0 Jul 22 02:34:57.092517 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.092398 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" event={"ID":"653d7521-ee37-4a47-b0ec-d8d52c53a971","Type":"ContainerDied","Data":"214b5006cdcd77265ec95da23261f869c455bd8a0c4f43412c18af246c91a395"} Jul 22 02:34:57.234611 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.234581 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:57.346656 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.346629 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle\") pod \"456c911f-b427-4e93-a6c8-09bdbbdce24b\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " Jul 22 02:34:57.346787 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.346680 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util\") pod \"456c911f-b427-4e93-a6c8-09bdbbdce24b\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " Jul 22 02:34:57.346787 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.346741 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrg5f\" (UniqueName: \"kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f\") pod \"456c911f-b427-4e93-a6c8-09bdbbdce24b\" (UID: \"456c911f-b427-4e93-a6c8-09bdbbdce24b\") " Jul 22 02:34:57.347191 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.347157 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle" (OuterVolumeSpecName: "bundle") pod "456c911f-b427-4e93-a6c8-09bdbbdce24b" (UID: "456c911f-b427-4e93-a6c8-09bdbbdce24b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:57.349005 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.348984 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f" (OuterVolumeSpecName: "kube-api-access-rrg5f") pod "456c911f-b427-4e93-a6c8-09bdbbdce24b" (UID: "456c911f-b427-4e93-a6c8-09bdbbdce24b"). InnerVolumeSpecName "kube-api-access-rrg5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:34:57.351796 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.351774 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util" (OuterVolumeSpecName: "util") pod "456c911f-b427-4e93-a6c8-09bdbbdce24b" (UID: "456c911f-b427-4e93-a6c8-09bdbbdce24b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:57.447292 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.447268 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrg5f\" (UniqueName: \"kubernetes.io/projected/456c911f-b427-4e93-a6c8-09bdbbdce24b-kube-api-access-rrg5f\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:57.447292 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.447293 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:57.447454 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:57.447303 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/456c911f-b427-4e93-a6c8-09bdbbdce24b-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:58.100082 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.100051 2549 generic.go:356] "Generic (PLEG): container finished" podID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerID="22961b12e9ca23735f9af68a790c61637e547e5165fc79e6f437550a9ce9c3f2" exitCode=0 Jul 22 02:34:58.100562 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.100139 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" event={"ID":"653d7521-ee37-4a47-b0ec-d8d52c53a971","Type":"ContainerDied","Data":"22961b12e9ca23735f9af68a790c61637e547e5165fc79e6f437550a9ce9c3f2"} Jul 22 02:34:58.102217 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.102193 2549 generic.go:356] "Generic (PLEG): container finished" podID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerID="4c576684c493361e25ed91d97bc4ecf7e9f1e10150b9a9ef597a1125b4e707a6" exitCode=0 Jul 22 02:34:58.102349 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.102272 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" event={"ID":"acc4d111-f49f-47b7-9c02-94523f4f880b","Type":"ContainerDied","Data":"4c576684c493361e25ed91d97bc4ecf7e9f1e10150b9a9ef597a1125b4e707a6"} Jul 22 02:34:58.104115 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.104097 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" Jul 22 02:34:58.104223 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.104094 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd" event={"ID":"456c911f-b427-4e93-a6c8-09bdbbdce24b","Type":"ContainerDied","Data":"9bf03ae6ef2ca14c1f0f62e8d543b85f5385562798da7a091405026514cf6300"} Jul 22 02:34:58.104223 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.104207 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bf03ae6ef2ca14c1f0f62e8d543b85f5385562798da7a091405026514cf6300" Jul 22 02:34:58.106578 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.106531 2549 generic.go:356] "Generic (PLEG): container finished" podID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerID="601584d600560dadb678fe1707be3ffc951e3ee5ecc6e2f9254675d5a49c35b3" exitCode=0 Jul 22 02:34:58.106686 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:58.106609 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" event={"ID":"1b4a216d-c4a3-499d-b410-b055eef44fc8","Type":"ContainerDied","Data":"601584d600560dadb678fe1707be3ffc951e3ee5ecc6e2f9254675d5a49c35b3"} Jul 22 02:34:59.262674 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.262644 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:34:59.299343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.299319 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:34:59.302212 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.302194 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:34:59.363294 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363230 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle\") pod \"acc4d111-f49f-47b7-9c02-94523f4f880b\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " Jul 22 02:34:59.363294 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363270 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p6ht\" (UniqueName: \"kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht\") pod \"653d7521-ee37-4a47-b0ec-d8d52c53a971\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " Jul 22 02:34:59.363501 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363320 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk2jp\" (UniqueName: \"kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp\") pod \"acc4d111-f49f-47b7-9c02-94523f4f880b\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " Jul 22 02:34:59.363501 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363341 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle\") pod \"653d7521-ee37-4a47-b0ec-d8d52c53a971\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " Jul 22 02:34:59.363501 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363386 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htprd\" (UniqueName: \"kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd\") pod \"1b4a216d-c4a3-499d-b410-b055eef44fc8\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " Jul 22 02:34:59.363501 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363428 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util\") pod \"acc4d111-f49f-47b7-9c02-94523f4f880b\" (UID: \"acc4d111-f49f-47b7-9c02-94523f4f880b\") " Jul 22 02:34:59.363501 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363457 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle\") pod \"1b4a216d-c4a3-499d-b410-b055eef44fc8\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " Jul 22 02:34:59.363748 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363675 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util\") pod \"653d7521-ee37-4a47-b0ec-d8d52c53a971\" (UID: \"653d7521-ee37-4a47-b0ec-d8d52c53a971\") " Jul 22 02:34:59.363748 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363716 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util\") pod \"1b4a216d-c4a3-499d-b410-b055eef44fc8\" (UID: \"1b4a216d-c4a3-499d-b410-b055eef44fc8\") " Jul 22 02:34:59.363958 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363933 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle" (OuterVolumeSpecName: "bundle") pod "1b4a216d-c4a3-499d-b410-b055eef44fc8" (UID: "1b4a216d-c4a3-499d-b410-b055eef44fc8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.364044 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.363976 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle" (OuterVolumeSpecName: "bundle") pod "653d7521-ee37-4a47-b0ec-d8d52c53a971" (UID: "653d7521-ee37-4a47-b0ec-d8d52c53a971"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.364044 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.364014 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.364696 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.364668 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle" (OuterVolumeSpecName: "bundle") pod "acc4d111-f49f-47b7-9c02-94523f4f880b" (UID: "acc4d111-f49f-47b7-9c02-94523f4f880b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.366140 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.366096 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht" (OuterVolumeSpecName: "kube-api-access-8p6ht") pod "653d7521-ee37-4a47-b0ec-d8d52c53a971" (UID: "653d7521-ee37-4a47-b0ec-d8d52c53a971"). InnerVolumeSpecName "kube-api-access-8p6ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:34:59.366252 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.366174 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp" (OuterVolumeSpecName: "kube-api-access-qk2jp") pod "acc4d111-f49f-47b7-9c02-94523f4f880b" (UID: "acc4d111-f49f-47b7-9c02-94523f4f880b"). InnerVolumeSpecName "kube-api-access-qk2jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:34:59.366632 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.366611 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd" (OuterVolumeSpecName: "kube-api-access-htprd") pod "1b4a216d-c4a3-499d-b410-b055eef44fc8" (UID: "1b4a216d-c4a3-499d-b410-b055eef44fc8"). InnerVolumeSpecName "kube-api-access-htprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:34:59.371126 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.371102 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util" (OuterVolumeSpecName: "util") pod "acc4d111-f49f-47b7-9c02-94523f4f880b" (UID: "acc4d111-f49f-47b7-9c02-94523f4f880b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.371250 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.371225 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util" (OuterVolumeSpecName: "util") pod "653d7521-ee37-4a47-b0ec-d8d52c53a971" (UID: "653d7521-ee37-4a47-b0ec-d8d52c53a971"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.371594 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.371563 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util" (OuterVolumeSpecName: "util") pod "1b4a216d-c4a3-499d-b410-b055eef44fc8" (UID: "1b4a216d-c4a3-499d-b410-b055eef44fc8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:34:59.464677 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464654 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6ht\" (UniqueName: \"kubernetes.io/projected/653d7521-ee37-4a47-b0ec-d8d52c53a971-kube-api-access-8p6ht\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464677 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464678 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2jp\" (UniqueName: \"kubernetes.io/projected/acc4d111-f49f-47b7-9c02-94523f4f880b-kube-api-access-qk2jp\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464688 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464697 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htprd\" (UniqueName: \"kubernetes.io/projected/1b4a216d-c4a3-499d-b410-b055eef44fc8-kube-api-access-htprd\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464706 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464713 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/653d7521-ee37-4a47-b0ec-d8d52c53a971-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464721 2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b4a216d-c4a3-499d-b410-b055eef44fc8-util\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:34:59.464803 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:34:59.464728 2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/acc4d111-f49f-47b7-9c02-94523f4f880b-bundle\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:35:00.117916 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.117889 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" Jul 22 02:35:00.118111 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.117889 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t" event={"ID":"653d7521-ee37-4a47-b0ec-d8d52c53a971","Type":"ContainerDied","Data":"9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af"} Jul 22 02:35:00.118111 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.118038 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0df8d95d4d3bbf0c0dbefe8835a455cc9db9e7e72a54a25ad8df2fb504a4af" Jul 22 02:35:00.119954 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.119930 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" Jul 22 02:35:00.120068 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.119976 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz" event={"ID":"acc4d111-f49f-47b7-9c02-94523f4f880b","Type":"ContainerDied","Data":"8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f"} Jul 22 02:35:00.120068 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.120053 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b4286f485a5c7dc49f709172705569523691de499f089cc82016d8da92d8c3f" Jul 22 02:35:00.122330 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.122311 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" Jul 22 02:35:00.122481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.122308 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f" event={"ID":"1b4a216d-c4a3-499d-b410-b055eef44fc8","Type":"ContainerDied","Data":"dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0"} Jul 22 02:35:00.122481 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:35:00.122449 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd7ec480085d8546d8fac73f3e49e6802abfc1c25ebcdad6748853cb2b6354b0" Jul 22 02:36:28.393893 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.393855 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394164 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394174 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394181 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394186 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394193 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394201 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394207 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394212 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394219 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394223 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394231 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394235 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394242 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394246 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="pull" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394254 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394258 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394263 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394268 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="util" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394273 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394278 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394286 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394291 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394302 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394307 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="extract" Jul 22 02:36:28.394343 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394353 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="acc4d111-f49f-47b7-9c02-94523f4f880b" containerName="extract" Jul 22 02:36:28.395071 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394359 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="456c911f-b427-4e93-a6c8-09bdbbdce24b" containerName="extract" Jul 22 02:36:28.395071 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394365 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="653d7521-ee37-4a47-b0ec-d8d52c53a971" containerName="extract" Jul 22 02:36:28.395071 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.394370 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4a216d-c4a3-499d-b410-b055eef44fc8" containerName="extract" Jul 22 02:36:28.397249 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.397231 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.400219 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.400200 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 02:36:28.401157 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.401138 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 02:36:28.401253 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.401141 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-srhxs\"" Jul 22 02:36:28.401253 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.401188 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 02:36:28.408906 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.408883 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:36:28.426560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.426532 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7348d98-514b-4875-a12b-de1d9cd807c5-config-file\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.426676 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.426571 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjvk\" (UniqueName: \"kubernetes.io/projected/a7348d98-514b-4875-a12b-de1d9cd807c5-kube-api-access-ncjvk\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.437985 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.437962 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:36:28.527976 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.527940 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7348d98-514b-4875-a12b-de1d9cd807c5-config-file\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.528133 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.527987 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjvk\" (UniqueName: \"kubernetes.io/projected/a7348d98-514b-4875-a12b-de1d9cd807c5-kube-api-access-ncjvk\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.528526 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.528508 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a7348d98-514b-4875-a12b-de1d9cd807c5-config-file\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.536358 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.536325 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjvk\" (UniqueName: \"kubernetes.io/projected/a7348d98-514b-4875-a12b-de1d9cd807c5-kube-api-access-ncjvk\") pod \"limitador-limitador-54d6d988d5-z2rxn\" (UID: \"a7348d98-514b-4875-a12b-de1d9cd807c5\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.708245 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.708206 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:28.856370 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:28.856349 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:36:28.858385 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:36:28.858361 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7348d98_514b_4875_a12b_de1d9cd807c5.slice/crio-87d7b055323b1d7415afaccc2789580b17d4470406dc516ab749dca86b281177 WatchSource:0}: Error finding container 87d7b055323b1d7415afaccc2789580b17d4470406dc516ab749dca86b281177: Status 404 returned error can't find the container with id 87d7b055323b1d7415afaccc2789580b17d4470406dc516ab749dca86b281177 Jul 22 02:36:29.559138 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:29.559095 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" event={"ID":"a7348d98-514b-4875-a12b-de1d9cd807c5","Type":"ContainerStarted","Data":"87d7b055323b1d7415afaccc2789580b17d4470406dc516ab749dca86b281177"} Jul 22 02:36:31.572025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:31.571935 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" event={"ID":"a7348d98-514b-4875-a12b-de1d9cd807c5","Type":"ContainerStarted","Data":"4678cefe4aaff61ebdd6cd243b424202e5c7da29e762d2436ef244dc1e587af5"} Jul 22 02:36:31.572025 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:31.572011 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:36:31.591132 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:31.591079 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" podStartSLOduration=1.157496508 podStartE2EDuration="3.591064568s" totalImagesPullingTime="2.43356806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:36:31.589890691 +0000 UTC m=+482.746902976" watchObservedRunningTime="2026-07-22 02:36:31.591064568 +0000 UTC m=+482.748076851" Jul 22 02:36:42.577992 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:36:42.577962 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-z2rxn" Jul 22 02:37:44.081074 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:37:44.080987 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:38:09.532958 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:09.532921 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:38:23.931153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:23.931112 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:38:41.539107 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:41.539064 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:38:53.220499 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.220423 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2"] Jul 22 02:38:53.223689 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.223671 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.250460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.250434 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 02:38:53.250603 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.250513 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 02:38:53.250603 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.250592 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pj5rv\"" Jul 22 02:38:53.250724 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.250630 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 02:38:53.257131 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.257110 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2"] Jul 22 02:38:53.373570 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373535 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.373570 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373579 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.373830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373608 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687a7e41-d64b-40ce-bc5a-9e14783044a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.373830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373646 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.373830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373707 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpvk\" (UniqueName: \"kubernetes.io/projected/687a7e41-d64b-40ce-bc5a-9e14783044a4-kube-api-access-hhpvk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.373830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.373764 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474577 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474493 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474735 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474598 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474735 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474622 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474735 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474651 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687a7e41-d64b-40ce-bc5a-9e14783044a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474735 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474693 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.474934 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.474738 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpvk\" (UniqueName: \"kubernetes.io/projected/687a7e41-d64b-40ce-bc5a-9e14783044a4-kube-api-access-hhpvk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.475137 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.475112 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.475228 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.475141 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.475228 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.475173 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.477047 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.477022 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/687a7e41-d64b-40ce-bc5a-9e14783044a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.477256 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.477238 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687a7e41-d64b-40ce-bc5a-9e14783044a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.486265 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.486235 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpvk\" (UniqueName: \"kubernetes.io/projected/687a7e41-d64b-40ce-bc5a-9e14783044a4-kube-api-access-hhpvk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2\" (UID: \"687a7e41-d64b-40ce-bc5a-9e14783044a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.533551 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.533527 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:38:53.667042 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.667014 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2"] Jul 22 02:38:53.668495 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:38:53.668471 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687a7e41_d64b_40ce_bc5a_9e14783044a4.slice/crio-be032cf2679923765d4232671bdbbed789ae8731c9de6a5f087168dd5421cd04 WatchSource:0}: Error finding container be032cf2679923765d4232671bdbbed789ae8731c9de6a5f087168dd5421cd04: Status 404 returned error can't find the container with id be032cf2679923765d4232671bdbbed789ae8731c9de6a5f087168dd5421cd04 Jul 22 02:38:53.669723 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:53.669708 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:38:54.269448 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:54.269399 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" event={"ID":"687a7e41-d64b-40ce-bc5a-9e14783044a4","Type":"ContainerStarted","Data":"be032cf2679923765d4232671bdbbed789ae8731c9de6a5f087168dd5421cd04"} Jul 22 02:38:55.031075 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:55.031041 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:38:55.275880 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:55.275843 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" event={"ID":"687a7e41-d64b-40ce-bc5a-9e14783044a4","Type":"ContainerStarted","Data":"71bcff3b14876fa2b51b5854e4587fe9a2be6e50018858a725da9fc0635d4e80"} Jul 22 02:38:55.295817 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:38:55.295716 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" podStartSLOduration=1.312679408 podStartE2EDuration="2.295700984s" totalImagesPullingTime="983.021576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:38:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:38:55.293271656 +0000 UTC m=+626.450283980" watchObservedRunningTime="2026-07-22 02:38:55.295700984 +0000 UTC m=+626.452713272" Jul 22 02:39:03.534032 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:03.533994 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:39:03.534533 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:03.534069 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:39:03.546485 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:03.546458 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:39:04.335986 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:04.335955 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2" Jul 22 02:39:15.029082 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:15.029039 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:39:30.533143 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:39:30.533109 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:41:13.233955 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:41:13.233911 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:41:22.629917 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:41:22.629882 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:41:34.020986 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:41:34.020949 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:42:51.707922 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.707884 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6765967b89-76trs"] Jul 22 02:42:51.711087 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.711068 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.714764 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.714742 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 02:42:51.714899 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.714787 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 02:42:51.714899 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.714742 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tkqdg\"" Jul 22 02:42:51.721339 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.721313 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6765967b89-76trs"] Jul 22 02:42:51.852718 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.852674 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zqb\" (UniqueName: \"kubernetes.io/projected/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-kube-api-access-x4zqb\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.852897 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.852737 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-webhook-cert\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.953455 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.953385 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zqb\" (UniqueName: \"kubernetes.io/projected/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-kube-api-access-x4zqb\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.953645 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.953480 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-webhook-cert\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.956033 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.956003 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-webhook-cert\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:51.963908 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:51.963851 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zqb\" (UniqueName: \"kubernetes.io/projected/d8db8daf-0355-47c0-95fb-0ecdaf1cfe19-kube-api-access-x4zqb\") pod \"maas-controller-6765967b89-76trs\" (UID: \"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19\") " pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:52.021621 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:52.021583 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:52.155729 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:52.155595 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6765967b89-76trs"] Jul 22 02:42:52.160945 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:42:52.160908 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8db8daf_0355_47c0_95fb_0ecdaf1cfe19.slice/crio-74dbe081f4719e7c077fc1a2364aaed4501dc90b751afa55e62bd66d004ee00f WatchSource:0}: Error finding container 74dbe081f4719e7c077fc1a2364aaed4501dc90b751afa55e62bd66d004ee00f: Status 404 returned error can't find the container with id 74dbe081f4719e7c077fc1a2364aaed4501dc90b751afa55e62bd66d004ee00f Jul 22 02:42:52.451341 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:52.451304 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6765967b89-76trs" event={"ID":"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19","Type":"ContainerStarted","Data":"74dbe081f4719e7c077fc1a2364aaed4501dc90b751afa55e62bd66d004ee00f"} Jul 22 02:42:55.470563 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:55.470521 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6765967b89-76trs" event={"ID":"d8db8daf-0355-47c0-95fb-0ecdaf1cfe19","Type":"ContainerStarted","Data":"ae38708a330ee95c76cd44fa5c71ff85c7977d601082e98f6bc59836052eb766"} Jul 22 02:42:55.471003 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:55.470656 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:42:55.489294 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:42:55.489243 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6765967b89-76trs" podStartSLOduration=2.238587236 podStartE2EDuration="4.489229847s" totalImagesPullingTime="2.250642611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:42:55.488256189 +0000 UTC m=+866.645268472" watchObservedRunningTime="2026-07-22 02:42:55.489229847 +0000 UTC m=+866.646242153" Jul 22 02:43:06.477558 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:43:06.477524 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6765967b89-76trs" Jul 22 02:43:17.427598 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:43:17.427552 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:43:26.017123 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:43:26.017046 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:00.025954 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:00.025914 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:09.628945 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:09.628907 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:18.531550 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:18.531517 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:30.025317 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:30.025283 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:41.438986 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:41.438947 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:44:51.031060 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:44:51.030937 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:45:02.422490 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:45:02.422451 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:45:12.632584 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:45:12.632543 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:45:35.132264 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:45:35.132224 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:45:44.733536 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:45:44.733500 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:46:09.120337 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:46:09.120300 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:46:19.344119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:46:19.344082 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:46:34.329274 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:46:34.329241 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:46:45.228793 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:46:45.228759 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:46:55.126007 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:46:55.125963 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:47:04.730268 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:47:04.730235 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:47:14.123592 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:47:14.123552 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:01.930745 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:01.930704 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:13.229940 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:13.229900 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:22.727444 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:22.727343 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:33.533931 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:33.533893 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:46.122223 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:46.122179 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:49:55.964452 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:49:55.964400 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:50:05.827583 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:50:05.827532 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:50:15.726894 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:50:15.726860 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:50:27.143092 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:50:27.143057 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:50:36.125348 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:50:36.125313 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:50:46.932466 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:50:46.932427 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:51:11.429180 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:11.429099 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:51:19.059586 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:19.059546 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:51:31.135000 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:31.134955 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:51:32.299094 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.299060 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:51:32.302446 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.302402 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.305744 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.305727 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d9228d76-kserve-self-signed-certs\"" Jul 22 02:51:32.315900 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.315875 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:51:32.423936 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.423898 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.424107 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.423947 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.424107 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.424089 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.424224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.424118 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfch5\" (UniqueName: \"kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.424224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.424148 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.424224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.424206 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525485 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525447 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525485 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525485 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfch5\" (UniqueName: \"kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525736 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525508 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525736 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525538 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525736 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525585 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525736 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525612 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525975 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525931 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.525975 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.525940 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.526113 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.526071 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.528203 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.528179 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.528322 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.528306 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.534038 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.534017 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfch5\" (UniqueName: \"kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5\") pod \"e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.612053 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.611966 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:32.746538 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:32.746502 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:51:32.748079 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:51:32.748052 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6e7136_c83a_44a8_bfb5_fd26b3bd2572.slice/crio-630947ba0c28aad938ca379d2406e2ae94af8a0935ec9d4722351f1434b2bf94 WatchSource:0}: Error finding container 630947ba0c28aad938ca379d2406e2ae94af8a0935ec9d4722351f1434b2bf94: Status 404 returned error can't find the container with id 630947ba0c28aad938ca379d2406e2ae94af8a0935ec9d4722351f1434b2bf94 Jul 22 02:51:33.031558 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:33.031510 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" event={"ID":"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572","Type":"ContainerStarted","Data":"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08"} Jul 22 02:51:33.031750 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:33.031566 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" event={"ID":"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572","Type":"ContainerStarted","Data":"630947ba0c28aad938ca379d2406e2ae94af8a0935ec9d4722351f1434b2bf94"} Jul 22 02:51:33.051235 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:33.051186 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" podStartSLOduration=1.051171268 podStartE2EDuration="1.051171268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:51:33.048943556 +0000 UTC m=+1384.205955840" watchObservedRunningTime="2026-07-22 02:51:33.051171268 +0000 UTC m=+1384.208183551" Jul 22 02:51:42.612871 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:42.612834 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:42.613323 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:42.612888 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:42.625442 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:42.625396 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:43.092777 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:43.092746 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:51:43.522246 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:43.522208 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:51:56.213467 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:56.213429 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:51:59.322561 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:51:59.322523 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:52:11.234830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.234765 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" podUID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" containerName="main" containerID="cri-o://03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08" gracePeriod=45 Jul 22 02:52:11.491492 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.491432 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:52:11.561714 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561685 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.561890 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561758 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.561890 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561791 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.561890 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561856 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.562055 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561899 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfch5\" (UniqueName: \"kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.562055 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561959 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir\") pod \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\" (UID: \"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572\") " Jul 22 02:52:11.562055 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.561992 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:52:11.562201 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.562188 2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:11.562201 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.562181 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home" (OuterVolumeSpecName: "home") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:52:11.562272 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.562206 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:52:11.564215 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.564187 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm" (OuterVolumeSpecName: "dshm") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:52:11.564323 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.564260 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 02:52:11.564363 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.564340 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5" (OuterVolumeSpecName: "kube-api-access-hfch5") pod "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" (UID: "aa6e7136-c83a-44a8-bfb5-fd26b3bd2572"). InnerVolumeSpecName "kube-api-access-hfch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:52:11.662901 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.662865 2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:11.662901 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.662904 2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:11.663119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.662919 2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:11.663119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.662934 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfch5\" (UniqueName: \"kubernetes.io/projected/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-kube-api-access-hfch5\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:11.663119 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:11.662950 2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:52:12.222967 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.222929 2549 generic.go:356] "Generic (PLEG): container finished" podID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" containerID="03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08" exitCode=0 Jul 22 02:52:12.223134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.222973 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" Jul 22 02:52:12.223134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.223003 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" event={"ID":"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572","Type":"ContainerDied","Data":"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08"} Jul 22 02:52:12.223134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.223042 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn" event={"ID":"aa6e7136-c83a-44a8-bfb5-fd26b3bd2572","Type":"ContainerDied","Data":"630947ba0c28aad938ca379d2406e2ae94af8a0935ec9d4722351f1434b2bf94"} Jul 22 02:52:12.223134 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.223064 2549 scope.go:117] "RemoveContainer" containerID="03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08" Jul 22 02:52:12.232116 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.232095 2549 scope.go:117] "RemoveContainer" containerID="03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08" Jul 22 02:52:12.232365 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:52:12.232349 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08\": container with ID starting with 03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08 not found: ID does not exist" containerID="03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08" Jul 22 02:52:12.232443 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.232370 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08"} err="failed to get container status \"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08\": rpc error: code = NotFound desc = could not find container \"03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08\": container with ID starting with 03f7bea5ee705c427c289c6559bc5276329b267fa8b2e9ea3f1eedf7458a7f08 not found: ID does not exist" Jul 22 02:52:12.245460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.245435 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:52:12.249627 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:12.249605 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d9228d76-kserve-7bf77677cf-phvdn"] Jul 22 02:52:13.379164 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:13.379123 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" path="/var/lib/kubelet/pods/aa6e7136-c83a-44a8-bfb5-fd26b3bd2572/volumes" Jul 22 02:52:13.425437 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:13.425395 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:52:20.355759 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:20.355685 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:52:36.636761 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:36.636726 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:52:50.842527 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:52:50.842490 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:53:00.928670 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:53:00.928627 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:53:15.045331 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:53:15.045287 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:53:25.748919 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:53:25.748875 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:53:45.046944 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:53:45.046896 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:54:03.032443 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:54:03.032393 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:54:16.638787 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:54:16.638752 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:54:25.922999 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:54:25.922961 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:54:38.028554 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:54:38.028516 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:54:49.827511 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:54:49.827473 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:55:01.528327 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:01.528290 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:55:16.333161 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:16.333125 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:55:29.825105 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:29.825035 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:55:35.836510 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.836473 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:55:35.836864 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.836811 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" containerName="main" Jul 22 02:55:35.836864 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.836821 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" containerName="main" Jul 22 02:55:35.836937 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.836884 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa6e7136-c83a-44a8-bfb5-fd26b3bd2572" containerName="main" Jul 22 02:55:35.839820 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.839800 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.842841 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.842821 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3c6d7b74-openshift-default-dockercfg-hsh4g\"" Jul 22 02:55:35.868283 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.868249 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:55:35.917979 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.917914 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918007 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918037 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918080 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnbd\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918186 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918113 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918396 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918183 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918396 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918275 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918396 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918304 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:35.918396 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:35.918331 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.018830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.018799 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019008 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.018880 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019008 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.018908 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019008 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.018932 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019008 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.018973 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019008 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019035 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019077 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnbd\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019239 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019111 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019263 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019334 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019604 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019560 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019666 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019638 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.019721 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.019701 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.021560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.021538 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.021877 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.021858 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.028132 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.028107 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.029215 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.029187 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnbd\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd\") pod \"e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.150105 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.150023 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:36.287350 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.287328 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:55:36.289259 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:55:36.289230 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a9c857_3ab8_44e6_8b05_5479dbdc18dd.slice/crio-763e52b37a40356cb953b807dffe00b6bf75f8e59af60bbb85b96e5229d150d2 WatchSource:0}: Error finding container 763e52b37a40356cb953b807dffe00b6bf75f8e59af60bbb85b96e5229d150d2: Status 404 returned error can't find the container with id 763e52b37a40356cb953b807dffe00b6bf75f8e59af60bbb85b96e5229d150d2 Jul 22 02:55:36.291374 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.291345 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:36.291437 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.291404 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:36.291490 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:36.291478 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:37.244717 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:37.244682 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerStarted","Data":"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781"} Jul 22 02:55:37.245109 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:37.244743 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerStarted","Data":"763e52b37a40356cb953b807dffe00b6bf75f8e59af60bbb85b96e5229d150d2"} Jul 22 02:55:37.267095 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:37.267051 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podStartSLOduration=2.267027745 podStartE2EDuration="2.267027745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:55:37.264505869 +0000 UTC m=+1628.421518144" watchObservedRunningTime="2026-07-22 02:55:37.267027745 +0000 UTC m=+1628.424040028" Jul 22 02:55:38.151262 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:38.151220 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:38.156168 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:38.156138 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:38.250925 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:38.250893 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:38.252046 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:38.252027 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:46.430034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:46.429997 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:55:54.132168 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.131868 2549 patch_prober.go:28] interesting pod/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:37774->10.133.0.34:15021: read: connection reset by peer" start-of-body= Jul 22 02:55:54.132168 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.131941 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": read tcp 10.133.0.2:37774->10.133.0.34:15021: read: connection reset by peer" Jul 22 02:55:54.332836 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.332748 2549 generic.go:356] "Generic (PLEG): container finished" podID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerID="3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781" exitCode=0 Jul 22 02:55:54.332836 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.332817 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerDied","Data":"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781"} Jul 22 02:55:54.333203 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.333191 2549 scope.go:117] "RemoveContainer" containerID="3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781" Jul 22 02:55:54.334211 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.334182 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:54.334324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.334254 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:54.334324 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:54.334286 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:55:55.342955 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:55.342918 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerStarted","Data":"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d"} Jul 22 02:55:56.150713 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:56.150671 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:56.150928 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:56.150905 2549 patch_prober.go:28] interesting pod/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 02:55:56.150974 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:56.150954 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 02:55:57.150963 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:57.150926 2549 patch_prober.go:28] interesting pod/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 02:55:57.151362 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:57.150988 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 02:55:58.150505 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:58.150468 2549 patch_prober.go:28] interesting pod/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 22 02:55:58.150699 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:58.150528 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 22 02:55:59.165399 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:59.165369 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:59.165847 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:59.165737 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:55:59.166515 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:55:59.166498 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:56:18.021255 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:18.021219 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:56:19.518858 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:19.518823 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:56:19.519271 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:19.519025 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" containerID="cri-o://4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d" gracePeriod=30 Jul 22 02:56:24.772063 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.772035 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:56:24.776214 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776192 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776312 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776224 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776312 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776271 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvnbd\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776312 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776294 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776332 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776360 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776460 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776406 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776604 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776483 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776604 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776513 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data\") pod \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\" (UID: \"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd\") " Jul 22 02:56:24.776704 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776618 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:56:24.776704 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776630 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:56:24.776830 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776810 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.776894 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776835 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.776894 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776839 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:56:24.777002 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.776955 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:56:24.777688 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.777628 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:56:24.778723 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.778697 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:56:24.778810 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.778726 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd" (OuterVolumeSpecName: "kube-api-access-kvnbd") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "kube-api-access-kvnbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:56:24.778952 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.778935 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:56:24.778999 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.778977 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" (UID: "d0a9c857-3ab8-44e6-8b05-5479dbdc18dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:56:24.878153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878120 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvnbd\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-kube-api-access-kvnbd\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878151 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878153 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878163 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878176 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878188 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878198 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:24.878394 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:24.878209 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:25.497468 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.497427 2549 generic.go:356] "Generic (PLEG): container finished" podID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerID="4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d" exitCode=0 Jul 22 02:56:25.497635 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.497476 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" Jul 22 02:56:25.497635 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.497514 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerDied","Data":"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d"} Jul 22 02:56:25.497635 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.497578 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n" event={"ID":"d0a9c857-3ab8-44e6-8b05-5479dbdc18dd","Type":"ContainerDied","Data":"763e52b37a40356cb953b807dffe00b6bf75f8e59af60bbb85b96e5229d150d2"} Jul 22 02:56:25.497635 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.497604 2549 scope.go:117] "RemoveContainer" containerID="4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d" Jul 22 02:56:25.506946 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.506924 2549 scope.go:117] "RemoveContainer" containerID="3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781" Jul 22 02:56:25.515375 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.515354 2549 scope.go:117] "RemoveContainer" containerID="4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d" Jul 22 02:56:25.515673 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:56:25.515649 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d\": container with ID starting with 4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d not found: ID does not exist" containerID="4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d" Jul 22 02:56:25.515789 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.515683 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d"} err="failed to get container status \"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d\": rpc error: code = NotFound desc = could not find container \"4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d\": container with ID starting with 4b14fad14ff8eca285c74911c20372642b2f66194ec11c771f9f846d5c67d97d not found: ID does not exist" Jul 22 02:56:25.515789 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.515708 2549 scope.go:117] "RemoveContainer" containerID="3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781" Jul 22 02:56:25.515954 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:56:25.515937 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781\": container with ID starting with 3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781 not found: ID does not exist" containerID="3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781" Jul 22 02:56:25.515992 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.515960 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781"} err="failed to get container status \"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781\": rpc error: code = NotFound desc = could not find container \"3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781\": container with ID starting with 3cf8068f0c0579410cdc5ca706a26ee73c1925318b129bc61c03b2e3fe4e8781 not found: ID does not exist" Jul 22 02:56:25.516565 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.516546 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:56:25.520790 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:25.520771 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3c6d7b74-openshift-default-7568dd54c5-tv49n"] Jul 22 02:56:27.378445 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:27.378397 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" path="/var/lib/kubelet/pods/d0a9c857-3ab8-44e6-8b05-5479dbdc18dd/volumes" Jul 22 02:56:29.721870 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:29.721786 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:56:48.435306 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.435266 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8"] Jul 22 02:56:48.435765 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.435679 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:56:48.435765 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.435694 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:56:48.435836 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.435771 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:56:48.440269 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.440251 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:48.443880 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.443854 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 02:56:48.444000 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.443855 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 02:56:48.446560 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.446523 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8"] Jul 22 02:56:48.588666 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.588632 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8\" (UID: \"b01681cd-b08f-4d11-ae4e-a79f7071884c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:48.689775 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.689681 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8\" (UID: \"b01681cd-b08f-4d11-ae4e-a79f7071884c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:48.690292 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.690271 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8\" (UID: \"b01681cd-b08f-4d11-ae4e-a79f7071884c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:48.752193 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.752143 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:48.889500 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.889474 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8"] Jul 22 02:56:48.891246 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:56:48.891220 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb01681cd_b08f_4d11_ae4e_a79f7071884c.slice/crio-c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380 WatchSource:0}: Error finding container c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380: Status 404 returned error can't find the container with id c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380 Jul 22 02:56:48.893506 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:48.893490 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 02:56:49.614161 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:49.614081 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" event={"ID":"b01681cd-b08f-4d11-ae4e-a79f7071884c","Type":"ContainerStarted","Data":"c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380"} Jul 22 02:56:52.632989 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:52.632948 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" event={"ID":"b01681cd-b08f-4d11-ae4e-a79f7071884c","Type":"ContainerStarted","Data":"0486ed53a8c784b04dac54a000a3cb9f208969b50a6f359924ae7406d7fffb79"} Jul 22 02:56:52.650624 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:52.650569 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" podStartSLOduration=1.600817888 podStartE2EDuration="4.650553292s" totalImagesPullingTime="3.049735404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:56:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:56:52.649097838 +0000 UTC m=+1703.806110122" watchObservedRunningTime="2026-07-22 02:56:52.650553292 +0000 UTC m=+1703.807565574" Jul 22 02:56:53.639090 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:53.639054 2549 generic.go:356] "Generic (PLEG): container finished" podID="b01681cd-b08f-4d11-ae4e-a79f7071884c" containerID="0486ed53a8c784b04dac54a000a3cb9f208969b50a6f359924ae7406d7fffb79" exitCode=0 Jul 22 02:56:53.639488 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:53.639137 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" event={"ID":"b01681cd-b08f-4d11-ae4e-a79f7071884c","Type":"ContainerDied","Data":"0486ed53a8c784b04dac54a000a3cb9f208969b50a6f359924ae7406d7fffb79"} Jul 22 02:56:54.774922 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:54.774891 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:54.846191 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:54.846161 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca\") pod \"b01681cd-b08f-4d11-ae4e-a79f7071884c\" (UID: \"b01681cd-b08f-4d11-ae4e-a79f7071884c\") " Jul 22 02:56:54.846586 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:54.846563 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b01681cd-b08f-4d11-ae4e-a79f7071884c" (UID: "b01681cd-b08f-4d11-ae4e-a79f7071884c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:56:54.947887 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:54.947858 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b01681cd-b08f-4d11-ae4e-a79f7071884c-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:56:55.650325 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:55.650251 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" Jul 22 02:56:55.650325 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:55.650289 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8" event={"ID":"b01681cd-b08f-4d11-ae4e-a79f7071884c","Type":"ContainerDied","Data":"c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380"} Jul 22 02:56:55.650522 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:56:55.650338 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9bd7144b561af0b0aa5a36aef905b50483eba71a07e8b92f55a05ae24654380" Jul 22 02:57:01.032053 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:01.032012 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:57:03.530689 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:03.530650 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8"] Jul 22 02:57:03.532669 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:03.532645 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-113f7429-jnnn8"] Jul 22 02:57:04.492664 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.492627 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:04.492991 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.492979 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01681cd-b08f-4d11-ae4e-a79f7071884c" containerName="revoke-keys" Jul 22 02:57:04.493034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.492995 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01681cd-b08f-4d11-ae4e-a79f7071884c" containerName="revoke-keys" Jul 22 02:57:04.493034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.493003 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:57:04.493034 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.493008 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:57:04.493126 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.493078 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a9c857-3ab8-44e6-8b05-5479dbdc18dd" containerName="istio-proxy" Jul 22 02:57:04.493126 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.493088 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01681cd-b08f-4d11-ae4e-a79f7071884c" containerName="revoke-keys" Jul 22 02:57:04.496066 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.496051 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.499504 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.499462 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-49fcd44d-openshift-default-dockercfg-7grhd\"" Jul 22 02:57:04.509889 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.509866 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:04.643687 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643645 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643702 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643794 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643822 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643844 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643886 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643984 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.643999 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.644166 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.644024 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trwgq\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745353 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745392 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745439 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trwgq\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745484 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745518 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745540 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745562 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745581 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.745756 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745613 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.746033 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745832 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.746033 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.745899 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.746146 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.746117 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.746146 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.746128 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.746390 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.746331 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.748091 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.748072 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.748185 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.748093 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.754094 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.754066 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trwgq\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.754278 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.754261 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token\") pod \"e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.806315 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.806282 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:04.950182 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:57:04.950152 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2394df_c1ac_49b6_beb5_b16c5354e36c.slice/crio-d1c3685d65aaee19b24ffee5d45b5d7a0cd63d173c83e559d069ed6617e4a1ca WatchSource:0}: Error finding container d1c3685d65aaee19b24ffee5d45b5d7a0cd63d173c83e559d069ed6617e4a1ca: Status 404 returned error can't find the container with id d1c3685d65aaee19b24ffee5d45b5d7a0cd63d173c83e559d069ed6617e4a1ca Jul 22 02:57:04.950476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.950458 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:04.952363 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.952331 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:04.952470 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.952404 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:04.952470 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:04.952465 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:05.380086 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:05.379995 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01681cd-b08f-4d11-ae4e-a79f7071884c" path="/var/lib/kubelet/pods/b01681cd-b08f-4d11-ae4e-a79f7071884c/volumes" Jul 22 02:57:05.704361 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:05.704327 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerStarted","Data":"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8"} Jul 22 02:57:05.704761 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:05.704440 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerStarted","Data":"d1c3685d65aaee19b24ffee5d45b5d7a0cd63d173c83e559d069ed6617e4a1ca"} Jul 22 02:57:05.725769 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:05.725709 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podStartSLOduration=1.72569251 podStartE2EDuration="1.72569251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:05.724400775 +0000 UTC m=+1716.881413094" watchObservedRunningTime="2026-07-22 02:57:05.72569251 +0000 UTC m=+1716.882704792" Jul 22 02:57:05.807228 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:05.807184 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:06.811935 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:06.811906 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:07.715782 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:07.715752 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:07.716781 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:07.716756 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:12.529250 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:12.529212 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:57:20.784094 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.784060 2549 generic.go:356] "Generic (PLEG): container finished" podID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerID="f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8" exitCode=0 Jul 22 02:57:20.784531 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.784099 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerDied","Data":"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8"} Jul 22 02:57:20.784531 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.784482 2549 scope.go:117] "RemoveContainer" containerID="f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8" Jul 22 02:57:20.785451 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.785403 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:20.785513 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.785491 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:20.785551 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:20.785522 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 02:57:21.790759 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:21.790722 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerStarted","Data":"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506"} Jul 22 02:57:21.806850 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:21.806832 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:21.807949 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:21.807924 2549 patch_prober.go:28] interesting pod/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 02:57:21.808054 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:21.807971 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 02:57:22.716168 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:22.716122 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:22.807708 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:22.807669 2549 patch_prober.go:28] interesting pod/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 02:57:22.808108 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:22.807726 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 02:57:23.806961 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:23.806929 2549 patch_prober.go:28] interesting pod/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 02:57:23.807141 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:23.806977 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 02:57:24.807002 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:24.806968 2549 patch_prober.go:28] interesting pod/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 02:57:24.807386 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:24.807024 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 02:57:25.806723 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:25.806687 2549 patch_prober.go:28] interesting pod/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 22 02:57:25.806910 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:25.806750 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 22 02:57:26.810517 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:26.810484 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:26.811674 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:26.811654 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:46.126476 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:46.126371 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:57:49.209641 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:49.209596 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:49.210715 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:49.210660 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" containerID="cri-o://abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506" gracePeriod=30 Jul 22 02:57:54.473172 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.473114 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:54.506640 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506609 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.506804 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506669 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.506804 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506713 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.506804 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506736 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.506804 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506785 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.507021 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506818 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.507021 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506856 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.507021 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506881 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.507021 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.506941 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trwgq\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq\") pod \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\" (UID: \"3e2394df-c1ac-49b6-beb5-b16c5354e36c\") " Jul 22 02:57:54.507224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507044 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:54.507224 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507159 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:57:54.507434 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507394 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:54.507678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507609 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.507678 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507635 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.507822 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.507800 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:54.508597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.508570 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:54.509673 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.509645 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 02:57:54.509760 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.509648 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:57:54.509804 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.509780 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 02:57:54.509863 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.509840 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq" (OuterVolumeSpecName: "kube-api-access-trwgq") pod "3e2394df-c1ac-49b6-beb5-b16c5354e36c" (UID: "3e2394df-c1ac-49b6-beb5-b16c5354e36c"). InnerVolumeSpecName "kube-api-access-trwgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 02:57:54.608823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608785 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.608823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608814 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.608823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608823 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.608823 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608833 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.609120 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608841 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.609120 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608849 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e2394df-c1ac-49b6-beb5-b16c5354e36c-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.609120 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.608857 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trwgq\" (UniqueName: \"kubernetes.io/projected/3e2394df-c1ac-49b6-beb5-b16c5354e36c-kube-api-access-trwgq\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:57:54.969386 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.969354 2549 generic.go:356] "Generic (PLEG): container finished" podID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerID="abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506" exitCode=0 Jul 22 02:57:54.969597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.969402 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" Jul 22 02:57:54.969597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.969443 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerDied","Data":"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506"} Jul 22 02:57:54.969597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.969535 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn" event={"ID":"3e2394df-c1ac-49b6-beb5-b16c5354e36c","Type":"ContainerDied","Data":"d1c3685d65aaee19b24ffee5d45b5d7a0cd63d173c83e559d069ed6617e4a1ca"} Jul 22 02:57:54.969597 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.969567 2549 scope.go:117] "RemoveContainer" containerID="abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506" Jul 22 02:57:54.978946 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.978932 2549 scope.go:117] "RemoveContainer" containerID="f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8" Jul 22 02:57:54.987038 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.987024 2549 scope.go:117] "RemoveContainer" containerID="abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506" Jul 22 02:57:54.987267 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:57:54.987249 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506\": container with ID starting with abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506 not found: ID does not exist" containerID="abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506" Jul 22 02:57:54.987304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.987275 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506"} err="failed to get container status \"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506\": rpc error: code = NotFound desc = could not find container \"abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506\": container with ID starting with abfaf11043a4c66c0fe93f0627f5bb1ee8431e0c689c516713f6aa17a55c9506 not found: ID does not exist" Jul 22 02:57:54.987304 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.987295 2549 scope.go:117] "RemoveContainer" containerID="f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8" Jul 22 02:57:54.987601 ip-10-0-133-125 kubenswrapper[2549]: E0722 02:57:54.987564 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8\": container with ID starting with f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8 not found: ID does not exist" containerID="f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8" Jul 22 02:57:54.987723 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.987671 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8"} err="failed to get container status \"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8\": rpc error: code = NotFound desc = could not find container \"f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8\": container with ID starting with f13359fbb96eb026c41863fe0f9eb0e0ee75d77cbb18c3ae7b438468949a60c8 not found: ID does not exist" Jul 22 02:57:54.990769 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.990747 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:54.997515 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:54.997494 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-49fcd44d-openshift-default-67bd445559-zxkpn"] Jul 22 02:57:55.379247 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:55.379176 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" path="/var/lib/kubelet/pods/3e2394df-c1ac-49b6-beb5-b16c5354e36c/volumes" Jul 22 02:57:57.324282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:57:57.324248 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:58:17.721132 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721099 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh"] Jul 22 02:58:17.721549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721447 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 02:58:17.721549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721460 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 02:58:17.721549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721476 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 02:58:17.721549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721482 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 02:58:17.721549 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.721539 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 02:58:17.724477 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.724461 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:17.726903 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.726882 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 02:58:17.726991 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.726882 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 02:58:17.731132 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.731080 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh"] Jul 22 02:58:17.811547 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.811497 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh\" (UID: \"8da07506-c65c-4275-8b37-bc2beb5be8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:17.912983 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.912945 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh\" (UID: \"8da07506-c65c-4275-8b37-bc2beb5be8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:17.913594 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:17.913573 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh\" (UID: \"8da07506-c65c-4275-8b37-bc2beb5be8ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:18.036531 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:18.036446 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:18.173282 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:18.173256 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh"] Jul 22 02:58:18.175747 ip-10-0-133-125 kubenswrapper[2549]: W0722 02:58:18.175717 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da07506_c65c_4275_8b37_bc2beb5be8ea.slice/crio-69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4 WatchSource:0}: Error finding container 69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4: Status 404 returned error can't find the container with id 69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4 Jul 22 02:58:19.098552 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:19.098467 2549 generic.go:356] "Generic (PLEG): container finished" podID="8da07506-c65c-4275-8b37-bc2beb5be8ea" containerID="cb8634b26bd0089d7296ef0021fc0c3d84dcffcb002923b8717301a3686e0cc2" exitCode=0 Jul 22 02:58:19.098914 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:19.098556 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" event={"ID":"8da07506-c65c-4275-8b37-bc2beb5be8ea","Type":"ContainerDied","Data":"cb8634b26bd0089d7296ef0021fc0c3d84dcffcb002923b8717301a3686e0cc2"} Jul 22 02:58:19.098914 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:19.098610 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" event={"ID":"8da07506-c65c-4275-8b37-bc2beb5be8ea","Type":"ContainerStarted","Data":"69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4"} Jul 22 02:58:20.234279 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:20.234250 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:20.332817 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:20.332783 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca\") pod \"8da07506-c65c-4275-8b37-bc2beb5be8ea\" (UID: \"8da07506-c65c-4275-8b37-bc2beb5be8ea\") " Jul 22 02:58:20.333102 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:20.333081 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8da07506-c65c-4275-8b37-bc2beb5be8ea" (UID: "8da07506-c65c-4275-8b37-bc2beb5be8ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 02:58:20.434562 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:20.434522 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da07506-c65c-4275-8b37-bc2beb5be8ea-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 02:58:21.112198 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:21.112167 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" Jul 22 02:58:21.112477 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:21.112203 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh" event={"ID":"8da07506-c65c-4275-8b37-bc2beb5be8ea","Type":"ContainerDied","Data":"69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4"} Jul 22 02:58:21.112477 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:21.112243 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69997d381317b89002fd72cefdc1c6fdbda8dec1b664237fce96a0d7913b6ca4" Jul 22 02:58:31.379226 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:31.379192 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 02:58:32.868140 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:32.868105 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh"] Jul 22 02:58:32.876254 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:32.876229 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e4d739e2-g6mrh"] Jul 22 02:58:33.378880 ip-10-0-133-125 kubenswrapper[2549]: I0722 02:58:33.378846 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da07506-c65c-4275-8b37-bc2beb5be8ea" path="/var/lib/kubelet/pods/8da07506-c65c-4275-8b37-bc2beb5be8ea/volumes" Jul 22 03:00:00.147125 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.147032 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7"] Jul 22 03:00:00.150711 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.150673 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da07506-c65c-4275-8b37-bc2beb5be8ea" containerName="revoke-keys" Jul 22 03:00:00.150711 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.150705 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da07506-c65c-4275-8b37-bc2beb5be8ea" containerName="revoke-keys" Jul 22 03:00:00.150889 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.150803 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2394df-c1ac-49b6-beb5-b16c5354e36c" containerName="istio-proxy" Jul 22 03:00:00.150889 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.150815 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da07506-c65c-4275-8b37-bc2beb5be8ea" containerName="revoke-keys" Jul 22 03:00:00.154308 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.154283 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7"] Jul 22 03:00:00.154464 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.154375 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" Jul 22 03:00:00.156846 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.156823 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 03:00:00.164398 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.164380 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" Jul 22 03:00:00.298528 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.298501 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7"] Jul 22 03:00:00.300563 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:00:00.300536 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa08604_39eb_48e4_b00e_b129ebda87e2.slice/crio-5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6 WatchSource:0}: Error finding container 5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6: Status 404 returned error can't find the container with id 5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6 Jul 22 03:00:00.609677 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.609644 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" event={"ID":"3aa08604-39eb-48e4-b00e-b129ebda87e2","Type":"ContainerStarted","Data":"ae548c82478fd39da90dd640afec5a0471f96eff3d7e7ac4d0aa9ecabdf2b262"} Jul 22 03:00:00.609857 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.609697 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" event={"ID":"3aa08604-39eb-48e4-b00e-b129ebda87e2","Type":"ContainerStarted","Data":"5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6"} Jul 22 03:00:00.624052 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:00.623996 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" podStartSLOduration=0.62397853 podStartE2EDuration="623.97853ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:00.623463495 +0000 UTC m=+1891.780475781" watchObservedRunningTime="2026-07-22 03:00:00.62397853 +0000 UTC m=+1891.780990808" Jul 22 03:00:01.140037 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:01.140000 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:00:01.616392 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:01.616360 2549 generic.go:356] "Generic (PLEG): container finished" podID="3aa08604-39eb-48e4-b00e-b129ebda87e2" containerID="ae548c82478fd39da90dd640afec5a0471f96eff3d7e7ac4d0aa9ecabdf2b262" exitCode=0 Jul 22 03:00:01.616819 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:01.616448 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" event={"ID":"3aa08604-39eb-48e4-b00e-b129ebda87e2","Type":"ContainerDied","Data":"ae548c82478fd39da90dd640afec5a0471f96eff3d7e7ac4d0aa9ecabdf2b262"} Jul 22 03:00:02.749796 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:02.749770 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" Jul 22 03:00:03.627890 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:03.627862 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" Jul 22 03:00:03.628058 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:03.627888 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744820-x8hz7" event={"ID":"3aa08604-39eb-48e4-b00e-b129ebda87e2","Type":"ContainerDied","Data":"5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6"} Jul 22 03:00:03.628058 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:03.627937 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5942561e10f5a05577087b4947f3215431b8bcdcdc26db67780c7d32c7d34eb6" Jul 22 03:00:09.535139 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:09.535101 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:00:23.540136 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:23.540098 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:00:31.548104 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.548063 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb"] Jul 22 03:00:31.548599 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.548426 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aa08604-39eb-48e4-b00e-b129ebda87e2" containerName="cleanup" Jul 22 03:00:31.548599 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.548437 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aa08604-39eb-48e4-b00e-b129ebda87e2" containerName="cleanup" Jul 22 03:00:31.548599 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.548494 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aa08604-39eb-48e4-b00e-b129ebda87e2" containerName="cleanup" Jul 22 03:00:31.551663 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.551644 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:31.554059 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.554035 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:00:31.554175 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.554074 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 03:00:31.560401 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.560378 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb"] Jul 22 03:00:31.606109 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.606080 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08162364-x27zb\" (UID: \"50619f24-8550-41b2-ad1d-b78bac3fbcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:31.706731 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.706697 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08162364-x27zb\" (UID: \"50619f24-8550-41b2-ad1d-b78bac3fbcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:31.707286 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.707268 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-08162364-x27zb\" (UID: \"50619f24-8550-41b2-ad1d-b78bac3fbcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:31.863790 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.863700 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:31.992680 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:31.992656 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb"] Jul 22 03:00:31.995404 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:00:31.995376 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50619f24_8550_41b2_ad1d_b78bac3fbcfa.slice/crio-4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872 WatchSource:0}: Error finding container 4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872: Status 404 returned error can't find the container with id 4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872 Jul 22 03:00:32.777659 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:32.777627 2549 generic.go:356] "Generic (PLEG): container finished" podID="50619f24-8550-41b2-ad1d-b78bac3fbcfa" containerID="fd7c1fc9b9e15f92b0de601d2783d2d932ad1453041f766a491bc58a098b248f" exitCode=0 Jul 22 03:00:32.778024 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:32.777688 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" event={"ID":"50619f24-8550-41b2-ad1d-b78bac3fbcfa","Type":"ContainerDied","Data":"fd7c1fc9b9e15f92b0de601d2783d2d932ad1453041f766a491bc58a098b248f"} Jul 22 03:00:32.778024 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:32.777748 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" event={"ID":"50619f24-8550-41b2-ad1d-b78bac3fbcfa","Type":"ContainerStarted","Data":"4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872"} Jul 22 03:00:33.917799 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:33.917766 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:34.028326 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.028291 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca\") pod \"50619f24-8550-41b2-ad1d-b78bac3fbcfa\" (UID: \"50619f24-8550-41b2-ad1d-b78bac3fbcfa\") " Jul 22 03:00:34.028769 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.028742 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50619f24-8550-41b2-ad1d-b78bac3fbcfa" (UID: "50619f24-8550-41b2-ad1d-b78bac3fbcfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:00:34.129813 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.129738 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50619f24-8550-41b2-ad1d-b78bac3fbcfa-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:00:34.789550 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.789510 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" event={"ID":"50619f24-8550-41b2-ad1d-b78bac3fbcfa","Type":"ContainerDied","Data":"4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872"} Jul 22 03:00:34.789550 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.789531 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb" Jul 22 03:00:34.789754 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:34.789560 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb0b95239b28701c9e78354833a59b2ca3609f51515966507a8de0d9657e872" Jul 22 03:00:45.124709 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:45.124674 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:00:46.625045 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:46.625011 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb"] Jul 22 03:00:46.629987 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:46.629961 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-08162364-x27zb"] Jul 22 03:00:47.378309 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:00:47.378267 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50619f24-8550-41b2-ad1d-b78bac3fbcfa" path="/var/lib/kubelet/pods/50619f24-8550-41b2-ad1d-b78bac3fbcfa/volumes" Jul 22 03:01:21.925182 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:21.925099 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:01:36.730126 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:36.730084 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:01:45.518126 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:45.518088 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:01:49.380771 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.380736 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf"] Jul 22 03:01:49.381135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.381051 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50619f24-8550-41b2-ad1d-b78bac3fbcfa" containerName="revoke-keys" Jul 22 03:01:49.381135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.381063 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="50619f24-8550-41b2-ad1d-b78bac3fbcfa" containerName="revoke-keys" Jul 22 03:01:49.381135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.381114 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="50619f24-8550-41b2-ad1d-b78bac3fbcfa" containerName="revoke-keys" Jul 22 03:01:49.383972 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.383955 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:49.386462 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.386439 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:01:49.386462 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.386450 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 03:01:49.391121 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.391097 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf"] Jul 22 03:01:49.475260 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.475223 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf\" (UID: \"81d4b7aa-d00b-4187-8bef-aa16d0fa302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:49.576891 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.576854 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf\" (UID: \"81d4b7aa-d00b-4187-8bef-aa16d0fa302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:49.577501 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.577481 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf\" (UID: \"81d4b7aa-d00b-4187-8bef-aa16d0fa302e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:49.695269 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.695234 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:49.832804 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:49.832778 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf"] Jul 22 03:01:49.835779 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:01:49.835748 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d4b7aa_d00b_4187_8bef_aa16d0fa302e.slice/crio-06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2 WatchSource:0}: Error finding container 06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2: Status 404 returned error can't find the container with id 06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2 Jul 22 03:01:50.173929 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:50.173888 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" event={"ID":"81d4b7aa-d00b-4187-8bef-aa16d0fa302e","Type":"ContainerStarted","Data":"a25a7da06756ac54f6f74bae8a41ab6f6488fcac437e98c5e1e3297d1aad86ba"} Jul 22 03:01:50.174145 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:50.173954 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" event={"ID":"81d4b7aa-d00b-4187-8bef-aa16d0fa302e","Type":"ContainerStarted","Data":"06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2"} Jul 22 03:01:50.192089 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:50.192032 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" podStartSLOduration=1.1920155590000001 podStartE2EDuration="1.192015559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:50.189495177 +0000 UTC m=+2001.346507463" watchObservedRunningTime="2026-07-22 03:01:50.192015559 +0000 UTC m=+2001.349027842" Jul 22 03:01:51.188637 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:51.188600 2549 generic.go:356] "Generic (PLEG): container finished" podID="81d4b7aa-d00b-4187-8bef-aa16d0fa302e" containerID="a25a7da06756ac54f6f74bae8a41ab6f6488fcac437e98c5e1e3297d1aad86ba" exitCode=0 Jul 22 03:01:51.189091 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:51.188688 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" event={"ID":"81d4b7aa-d00b-4187-8bef-aa16d0fa302e","Type":"ContainerDied","Data":"a25a7da06756ac54f6f74bae8a41ab6f6488fcac437e98c5e1e3297d1aad86ba"} Jul 22 03:01:52.330321 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:52.330294 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:52.502889 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:52.502789 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca\") pod \"81d4b7aa-d00b-4187-8bef-aa16d0fa302e\" (UID: \"81d4b7aa-d00b-4187-8bef-aa16d0fa302e\") " Jul 22 03:01:52.503133 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:52.503111 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81d4b7aa-d00b-4187-8bef-aa16d0fa302e" (UID: "81d4b7aa-d00b-4187-8bef-aa16d0fa302e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:01:52.604135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:52.604095 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d4b7aa-d00b-4187-8bef-aa16d0fa302e-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:01:53.200582 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:53.200550 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" Jul 22 03:01:53.200782 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:53.200554 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf" event={"ID":"81d4b7aa-d00b-4187-8bef-aa16d0fa302e","Type":"ContainerDied","Data":"06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2"} Jul 22 03:01:53.200782 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:01:53.200671 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ef6397189a71d2385079fdd32a4433895c078db3de990e3d221faaa1f8c7f2" Jul 22 03:02:04.356658 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:02:04.356615 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:02:09.470385 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:02:09.470343 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf"] Jul 22 03:02:09.472913 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:02:09.472889 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ee84bc3c-tncjf"] Jul 22 03:02:11.379096 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:02:11.379061 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d4b7aa-d00b-4187-8bef-aa16d0fa302e" path="/var/lib/kubelet/pods/81d4b7aa-d00b-4187-8bef-aa16d0fa302e/volumes" Jul 22 03:02:13.220866 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:02:13.220825 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:03:04.228542 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:03:04.228441 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:03:17.274313 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:03:17.274273 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:03:29.489535 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:03:29.489505 2549 scope.go:117] "RemoveContainer" containerID="0486ed53a8c784b04dac54a000a3cb9f208969b50a6f359924ae7406d7fffb79" Jul 22 03:04:03.931879 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:03.931840 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:04:04.664972 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.664937 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:04:04.665371 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.665354 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d4b7aa-d00b-4187-8bef-aa16d0fa302e" containerName="revoke-keys" Jul 22 03:04:04.665498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.665375 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d4b7aa-d00b-4187-8bef-aa16d0fa302e" containerName="revoke-keys" Jul 22 03:04:04.665498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.665483 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d4b7aa-d00b-4187-8bef-aa16d0fa302e" containerName="revoke-keys" Jul 22 03:04:04.668680 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.668657 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.671176 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.671138 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-caf067-openshift-default-dockercfg-5bdxg\"" Jul 22 03:04:04.695832 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.695795 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:04:04.746862 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746829 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.746862 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746867 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747051 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746892 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747051 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746906 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747051 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746922 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gntb\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747051 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746947 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747051 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.746978 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747207 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.747055 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.747207 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.747102 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848322 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848277 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848322 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848326 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848486 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848553 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848605 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848660 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848685 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848713 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gntb\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848717 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.848773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848751 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.849006 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848771 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.849006 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848785 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.849006 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.848946 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.849312 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.849294 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.851094 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.851075 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.851334 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.851313 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.857165 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.857137 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.857256 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.857212 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gntb\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb\") pod \"e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:04.993343 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:04.993307 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:05.133577 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.133551 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:04:05.135059 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:04:05.135034 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d45969_7f77_4bb5_9a97_ffed0c0f7475.slice/crio-3a45bf3f51485e1c838151fcf6ce9036067de599f4b9d624bd3812b2a9049b03 WatchSource:0}: Error finding container 3a45bf3f51485e1c838151fcf6ce9036067de599f4b9d624bd3812b2a9049b03: Status 404 returned error can't find the container with id 3a45bf3f51485e1c838151fcf6ce9036067de599f4b9d624bd3812b2a9049b03 Jul 22 03:04:05.137134 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.137103 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:05.137228 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.137168 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:05.137228 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.137194 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:05.901819 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.901779 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerStarted","Data":"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce"} Jul 22 03:04:05.902029 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.901841 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerStarted","Data":"3a45bf3f51485e1c838151fcf6ce9036067de599f4b9d624bd3812b2a9049b03"} Jul 22 03:04:05.928940 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.928881 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podStartSLOduration=1.928864399 podStartE2EDuration="1.928864399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:05.926815622 +0000 UTC m=+2137.083827904" watchObservedRunningTime="2026-07-22 03:04:05.928864399 +0000 UTC m=+2137.085876682" Jul 22 03:04:05.993948 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:05.993901 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:06.998583 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:06.998551 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:07.913289 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:07.913256 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:07.914101 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:07.914080 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:20.982469 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.982432 2549 generic.go:356] "Generic (PLEG): container finished" podID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerID="281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce" exitCode=0 Jul 22 03:04:20.982958 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.982481 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerDied","Data":"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce"} Jul 22 03:04:20.982958 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.982878 2549 scope.go:117] "RemoveContainer" containerID="281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce" Jul 22 03:04:20.983997 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.983972 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:20.984096 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.984039 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:20.984096 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:20.984081 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:04:21.284089 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:21.283977 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:04:21.990454 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:21.990400 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerStarted","Data":"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9"} Jul 22 03:04:21.994428 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:21.994390 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:22.913274 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:22.913240 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:22.995192 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:22.995156 2549 patch_prober.go:28] interesting pod/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 03:04:22.995684 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:22.995233 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 03:04:23.993945 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:23.993902 2549 patch_prober.go:28] interesting pod/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 03:04:23.994139 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:23.993977 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 03:04:24.994095 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:24.994054 2549 patch_prober.go:28] interesting pod/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 22 03:04:24.994509 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:24.994115 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 22 03:04:25.998202 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:25.998171 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:25.999217 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:25.999196 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:04:29.510767 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:29.510736 2549 scope.go:117] "RemoveContainer" containerID="cb8634b26bd0089d7296ef0021fc0c3d84dcffcb002923b8717301a3686e0cc2" Jul 22 03:04:32.946372 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:32.946335 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:04:56.432025 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.431993 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:04:56.435498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.435478 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.438164 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.438145 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-caf067\"/\"test-model-caf067-kserve-self-signed-certs\"" Jul 22 03:04:56.439015 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.439000 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-caf067\"/\"default-dockercfg-z58ql\"" Jul 22 03:04:56.441199 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.441176 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-caf067\"/\"kube-root-ca.crt\"" Jul 22 03:04:56.442005 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.441984 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-caf067\"/\"openshift-service-ca.crt\"" Jul 22 03:04:56.459118 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.459090 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:04:56.517374 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517342 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.517374 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517376 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.517645 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517403 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.517645 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517483 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.517645 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517535 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhtn\" (UniqueName: \"kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.517645 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.517564 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618099 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618067 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618099 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618102 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618126 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618153 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618201 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhtn\" (UniqueName: \"kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618235 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618528 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618511 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618585 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618545 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.618668 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.618649 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.620573 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.620546 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.620746 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.620727 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.629995 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.629972 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhtn\" (UniqueName: \"kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn\") pod \"test-model-caf067-kserve-747998b987-mlcvh\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.745089 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.745000 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:04:56.909166 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:56.909118 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:04:56.916774 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:04:56.916744 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5565f08b_701b_4289_944c_d7b14794ab59.slice/crio-419be6c50c05be5e4272a7bd7f916bb9fdf31665085efffb64f2cd1e21f42c46 WatchSource:0}: Error finding container 419be6c50c05be5e4272a7bd7f916bb9fdf31665085efffb64f2cd1e21f42c46: Status 404 returned error can't find the container with id 419be6c50c05be5e4272a7bd7f916bb9fdf31665085efffb64f2cd1e21f42c46 Jul 22 03:04:57.172725 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:57.172692 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" event={"ID":"5565f08b-701b-4289-944c-d7b14794ab59","Type":"ContainerStarted","Data":"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b"} Jul 22 03:04:57.172896 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:04:57.172737 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" event={"ID":"5565f08b-701b-4289-944c-d7b14794ab59","Type":"ContainerStarted","Data":"419be6c50c05be5e4272a7bd7f916bb9fdf31665085efffb64f2cd1e21f42c46"} Jul 22 03:05:00.947111 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:00.947047 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" podStartSLOduration=4.947026898 podStartE2EDuration="4.947026898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:57.215284561 +0000 UTC m=+2188.372296869" watchObservedRunningTime="2026-07-22 03:05:00.947026898 +0000 UTC m=+2192.104039181" Jul 22 03:05:00.947866 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:00.947843 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:05:06.745515 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:06.745478 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:06.746005 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:06.745543 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:06.758349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:06.758318 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:07.236646 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:07.236617 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:14.845399 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:14.845360 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:05:38.496361 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.496328 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:05:38.613474 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.613406 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw"] Jul 22 03:05:38.616832 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.616805 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:38.619715 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.619690 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 03:05:38.619965 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.619952 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 03:05:38.630751 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.630725 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw"] Jul 22 03:05:38.696772 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.696736 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-317968-pz7kw\" (UID: \"2eb8745a-e995-4133-a6f1-93f5cb96451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:38.798242 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.798147 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-317968-pz7kw\" (UID: \"2eb8745a-e995-4133-a6f1-93f5cb96451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:38.798801 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.798783 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-317968-pz7kw\" (UID: \"2eb8745a-e995-4133-a6f1-93f5cb96451b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:38.927247 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:38.927204 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:39.072883 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:39.072850 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw"] Jul 22 03:05:39.073525 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:05:39.073497 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb8745a_e995_4133_a6f1_93f5cb96451b.slice/crio-dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602 WatchSource:0}: Error finding container dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602: Status 404 returned error can't find the container with id dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602 Jul 22 03:05:39.392319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:39.392272 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" event={"ID":"2eb8745a-e995-4133-a6f1-93f5cb96451b","Type":"ContainerStarted","Data":"3d28d0c5a2102ceabad074cb0a884ab9e78332053756290bbe06f0c97b1208d1"} Jul 22 03:05:39.392517 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:39.392331 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" event={"ID":"2eb8745a-e995-4133-a6f1-93f5cb96451b","Type":"ContainerStarted","Data":"dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602"} Jul 22 03:05:39.422012 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:39.421953 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" podStartSLOduration=1.421937497 podStartE2EDuration="1.421937497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:05:39.417708369 +0000 UTC m=+2230.574720652" watchObservedRunningTime="2026-07-22 03:05:39.421937497 +0000 UTC m=+2230.578949780" Jul 22 03:05:40.399237 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:40.399201 2549 generic.go:356] "Generic (PLEG): container finished" podID="2eb8745a-e995-4133-a6f1-93f5cb96451b" containerID="3d28d0c5a2102ceabad074cb0a884ab9e78332053756290bbe06f0c97b1208d1" exitCode=0 Jul 22 03:05:40.399718 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:40.399256 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" event={"ID":"2eb8745a-e995-4133-a6f1-93f5cb96451b","Type":"ContainerDied","Data":"3d28d0c5a2102ceabad074cb0a884ab9e78332053756290bbe06f0c97b1208d1"} Jul 22 03:05:41.537280 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:41.537254 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:41.627546 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:41.627505 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca\") pod \"2eb8745a-e995-4133-a6f1-93f5cb96451b\" (UID: \"2eb8745a-e995-4133-a6f1-93f5cb96451b\") " Jul 22 03:05:41.627879 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:41.627853 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eb8745a-e995-4133-a6f1-93f5cb96451b" (UID: "2eb8745a-e995-4133-a6f1-93f5cb96451b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:05:41.729017 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:41.728977 2549 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb8745a-e995-4133-a6f1-93f5cb96451b-maas-api-service-ca\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:42.410964 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:42.410939 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" Jul 22 03:05:42.411141 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:42.410943 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw" event={"ID":"2eb8745a-e995-4133-a6f1-93f5cb96451b","Type":"ContainerDied","Data":"dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602"} Jul 22 03:05:42.411141 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:42.411056 2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dff31c9c6da6fb62e95e23315af8e01b9522894fa944e230443e80a54b824602" Jul 22 03:05:44.636784 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:44.636743 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:05:53.516826 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.516740 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" podUID="5565f08b-701b-4289-944c-d7b14794ab59" containerName="main" containerID="cri-o://ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b" gracePeriod=45 Jul 22 03:05:53.726212 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.726177 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw"] Jul 22 03:05:53.730937 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.730909 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-317968-pz7kw"] Jul 22 03:05:53.783723 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.783699 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:53.938457 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938403 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938478 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938521 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938569 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938628 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzhtn\" (UniqueName: \"kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938921 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938663 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm\") pod \"5565f08b-701b-4289-944c-d7b14794ab59\" (UID: \"5565f08b-701b-4289-944c-d7b14794ab59\") " Jul 22 03:05:53.938921 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938730 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:53.938921 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938906 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache" (OuterVolumeSpecName: "model-cache") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:53.939055 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938920 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home" (OuterVolumeSpecName: "home") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:53.939055 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.938959 2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:53.940955 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.940923 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:05:53.941065 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.940991 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm" (OuterVolumeSpecName: "dshm") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:05:53.941065 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:53.941000 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn" (OuterVolumeSpecName: "kube-api-access-fzhtn") pod "5565f08b-701b-4289-944c-d7b14794ab59" (UID: "5565f08b-701b-4289-944c-d7b14794ab59"). InnerVolumeSpecName "kube-api-access-fzhtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:05:54.039970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.039882 2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:54.039970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.039914 2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5565f08b-701b-4289-944c-d7b14794ab59-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:54.039970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.039924 2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:54.039970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.039932 2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5565f08b-701b-4289-944c-d7b14794ab59-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:54.039970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.039940 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhtn\" (UniqueName: \"kubernetes.io/projected/5565f08b-701b-4289-944c-d7b14794ab59-kube-api-access-fzhtn\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:05:54.473477 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.473442 2549 generic.go:356] "Generic (PLEG): container finished" podID="5565f08b-701b-4289-944c-d7b14794ab59" containerID="ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b" exitCode=0 Jul 22 03:05:54.473651 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.473482 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" Jul 22 03:05:54.473651 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.473518 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" event={"ID":"5565f08b-701b-4289-944c-d7b14794ab59","Type":"ContainerDied","Data":"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b"} Jul 22 03:05:54.473651 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.473570 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh" event={"ID":"5565f08b-701b-4289-944c-d7b14794ab59","Type":"ContainerDied","Data":"419be6c50c05be5e4272a7bd7f916bb9fdf31665085efffb64f2cd1e21f42c46"} Jul 22 03:05:54.473651 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.473587 2549 scope.go:117] "RemoveContainer" containerID="ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b" Jul 22 03:05:54.483319 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.483302 2549 scope.go:117] "RemoveContainer" containerID="ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b" Jul 22 03:05:54.483630 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:05:54.483611 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b\": container with ID starting with ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b not found: ID does not exist" containerID="ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b" Jul 22 03:05:54.483684 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.483640 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b"} err="failed to get container status \"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b\": rpc error: code = NotFound desc = could not find container \"ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b\": container with ID starting with ee9a4b0335481e3dee94d949705a061a2f93960f8fe8db8c9bbfc14e05ea666b not found: ID does not exist" Jul 22 03:05:54.496065 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.496035 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:05:54.500004 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.499982 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-caf067/test-model-caf067-kserve-747998b987-mlcvh"] Jul 22 03:05:54.742477 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:54.742384 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:05:55.379619 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:55.379588 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb8745a-e995-4133-a6f1-93f5cb96451b" path="/var/lib/kubelet/pods/2eb8745a-e995-4133-a6f1-93f5cb96451b/volumes" Jul 22 03:05:55.379979 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:05:55.379966 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5565f08b-701b-4289-944c-d7b14794ab59" path="/var/lib/kubelet/pods/5565f08b-701b-4289-944c-d7b14794ab59/volumes" Jul 22 03:06:15.463303 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:15.463267 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:06:24.190040 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:24.190003 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:06:24.190588 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:24.190203 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" containerID="cri-o://0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9" gracePeriod=30 Jul 22 03:06:26.001135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:26.001092 2549 patch_prober.go:28] interesting pod/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 03:06:26.001544 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:26.001156 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 03:06:29.450029 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.450005 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:06:29.545338 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545249 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545338 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545319 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545355 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545387 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545465 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545510 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545573 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gntb\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545604 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.545699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.545643 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token\") pod \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\" (UID: \"e3d45969-7f77-4bb5-9a97-ffed0c0f7475\") " Jul 22 03:06:29.546576 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.546122 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:29.546576 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.546335 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:29.546576 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.546547 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:29.546814 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.546579 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:29.547277 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.547255 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:06:29.549207 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.549175 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:06:29.549313 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.549296 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb" (OuterVolumeSpecName: "kube-api-access-8gntb") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "kube-api-access-8gntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:29.549775 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.549746 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:06:29.550139 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.550113 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3d45969-7f77-4bb5-9a97-ffed0c0f7475" (UID: "e3d45969-7f77-4bb5-9a97-ffed0c0f7475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:06:29.646884 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646851 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.646884 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646880 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.646884 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646889 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646897 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646906 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646915 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gntb\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-kube-api-access-8gntb\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646924 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646932 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.647115 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.646942 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d45969-7f77-4bb5-9a97-ffed0c0f7475-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:06:29.661910 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.661877 2549 generic.go:356] "Generic (PLEG): container finished" podID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerID="0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9" exitCode=0 Jul 22 03:06:29.662062 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.661923 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" Jul 22 03:06:29.662062 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.661923 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerDied","Data":"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9"} Jul 22 03:06:29.662062 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.662041 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk" event={"ID":"e3d45969-7f77-4bb5-9a97-ffed0c0f7475","Type":"ContainerDied","Data":"3a45bf3f51485e1c838151fcf6ce9036067de599f4b9d624bd3812b2a9049b03"} Jul 22 03:06:29.662208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.662072 2549 scope.go:117] "RemoveContainer" containerID="0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9" Jul 22 03:06:29.671208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.671190 2549 scope.go:117] "RemoveContainer" containerID="281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce" Jul 22 03:06:29.680877 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.680858 2549 scope.go:117] "RemoveContainer" containerID="0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9" Jul 22 03:06:29.681168 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:06:29.681146 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9\": container with ID starting with 0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9 not found: ID does not exist" containerID="0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9" Jul 22 03:06:29.681233 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.681177 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9"} err="failed to get container status \"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9\": rpc error: code = NotFound desc = could not find container \"0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9\": container with ID starting with 0e2df45b9eac58b2992bcbcefc35e5b201b80991f944203cfefb0d9bc1cc65f9 not found: ID does not exist" Jul 22 03:06:29.681233 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.681197 2549 scope.go:117] "RemoveContainer" containerID="281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce" Jul 22 03:06:29.681493 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:06:29.681462 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce\": container with ID starting with 281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce not found: ID does not exist" containerID="281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce" Jul 22 03:06:29.681594 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.681499 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce"} err="failed to get container status \"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce\": rpc error: code = NotFound desc = could not find container \"281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce\": container with ID starting with 281c54623467292c38c0c368577fda526a1aa986167e80e98035a938cd75bcce not found: ID does not exist" Jul 22 03:06:29.708498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.708466 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:06:29.725020 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:29.724991 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-caf067-openshift-default-cf6b989d9-wz4hk"] Jul 22 03:06:31.378397 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:31.378361 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" path="/var/lib/kubelet/pods/e3d45969-7f77-4bb5-9a97-ffed0c0f7475/volumes" Jul 22 03:06:33.772808 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.772772 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:06:33.773272 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773252 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5565f08b-701b-4289-944c-d7b14794ab59" containerName="main" Jul 22 03:06:33.773349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773277 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="5565f08b-701b-4289-944c-d7b14794ab59" containerName="main" Jul 22 03:06:33.773349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773302 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb8745a-e995-4133-a6f1-93f5cb96451b" containerName="revoke-keys" Jul 22 03:06:33.773349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773311 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb8745a-e995-4133-a6f1-93f5cb96451b" containerName="revoke-keys" Jul 22 03:06:33.773349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773330 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773349 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773339 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773352 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773362 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773484 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb8745a-e995-4133-a6f1-93f5cb96451b" containerName="revoke-keys" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773500 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773510 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d45969-7f77-4bb5-9a97-ffed0c0f7475" containerName="istio-proxy" Jul 22 03:06:33.773621 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.773520 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="5565f08b-701b-4289-944c-d7b14794ab59" containerName="main" Jul 22 03:06:33.776775 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.776752 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.779685 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.779664 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f3887a-openshift-default-dockercfg-8s8t2\"" Jul 22 03:06:33.803451 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.803403 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:06:33.883662 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883620 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.883855 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883694 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvmh2\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.883855 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883746 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.883855 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883791 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.883855 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883827 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.883855 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883857 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.884074 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883878 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.884074 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883893 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.884074 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.883942 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985111 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985069 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvmh2\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985143 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985167 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985192 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985223 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985253 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985274 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985674 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985622 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985674 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985655 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985781 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985732 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985781 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985741 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985917 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985334 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.985974 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.985957 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.986073 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.986050 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.987656 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.987630 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:33.988269 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:33.988252 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:34.005761 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.005734 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:34.011019 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.010993 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvmh2\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2\") pod \"e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:34.086961 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.086861 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:34.230066 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.230031 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:06:34.230205 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:06:34.230186 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5dffc5_595b_4075_a1c7_ba2e9f58e9d2.slice/crio-17210857b11954a7408075f3725068cbbe05b5e6381f6e47b0a5487829a5c44c WatchSource:0}: Error finding container 17210857b11954a7408075f3725068cbbe05b5e6381f6e47b0a5487829a5c44c: Status 404 returned error can't find the container with id 17210857b11954a7408075f3725068cbbe05b5e6381f6e47b0a5487829a5c44c Jul 22 03:06:34.232571 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.232539 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:34.232648 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.232606 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:34.232648 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.232638 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:34.362722 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.362640 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:06:34.691161 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.691125 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerStarted","Data":"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90"} Jul 22 03:06:34.691339 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.691190 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerStarted","Data":"17210857b11954a7408075f3725068cbbe05b5e6381f6e47b0a5487829a5c44c"} Jul 22 03:06:34.724961 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:34.724899 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podStartSLOduration=1.724878805 podStartE2EDuration="1.724878805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:34.722733799 +0000 UTC m=+2285.879746085" watchObservedRunningTime="2026-07-22 03:06:34.724878805 +0000 UTC m=+2285.881891088" Jul 22 03:06:35.087757 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:35.087670 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:35.092985 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:35.092959 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:35.696426 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:35.696385 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:35.697329 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:35.697308 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:49.775075 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.775043 2549 generic.go:356] "Generic (PLEG): container finished" podID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerID="1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90" exitCode=0 Jul 22 03:06:49.775523 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.775115 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerDied","Data":"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90"} Jul 22 03:06:49.775523 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.775504 2549 scope.go:117] "RemoveContainer" containerID="1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90" Jul 22 03:06:49.776555 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.776529 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:49.776631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.776597 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:49.776631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:49.776626 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:06:50.696469 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:50.696432 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:50.736029 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:50.735987 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:06:50.783018 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:50.782976 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerStarted","Data":"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b"} Jul 22 03:06:51.087358 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:51.087264 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:51.088378 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:51.088355 2549 patch_prober.go:28] interesting pod/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 03:06:51.088498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:51.088430 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 03:06:52.087746 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:52.087709 2549 patch_prober.go:28] interesting pod/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 03:06:52.088202 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:52.087782 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 03:06:53.087514 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:53.087477 2549 patch_prober.go:28] interesting pod/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 03:06:53.087701 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:53.087542 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 03:06:54.087980 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:54.087941 2549 patch_prober.go:28] interesting pod/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 22 03:06:54.088373 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:54.088007 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 22 03:06:55.091645 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:55.091615 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:55.092125 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:55.091916 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:06:55.092742 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:06:55.092726 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:07:08.732513 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:08.732477 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:07:29.562959 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:29.562870 2549 scope.go:117] "RemoveContainer" containerID="fd7c1fc9b9e15f92b0de601d2783d2d932ad1453041f766a491bc58a098b248f" Jul 22 03:07:36.429880 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:36.429846 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:07:43.328383 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:43.328349 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:07:43.328815 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:43.328579 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" containerID="cri-o://b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b" gracePeriod=30 Jul 22 03:07:48.581028 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.581000 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:07:48.774246 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774212 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774438 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774287 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774438 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774327 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774438 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774360 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvmh2\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774438 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774399 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774612 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774475 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774612 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774498 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774612 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774522 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:48.774612 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774539 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774612 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774570 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs\") pod \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\" (UID: \"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2\") " Jul 22 03:07:48.774869 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774756 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:48.774945 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774923 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.775011 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774942 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:48.775011 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.774952 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.775011 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.775002 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:07:48.775390 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.775324 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data" (OuterVolumeSpecName: "istio-data") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:48.776897 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.776857 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:07:48.776991 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.776935 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2" (OuterVolumeSpecName: "kube-api-access-zvmh2") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "kube-api-access-zvmh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:07:48.776991 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.776936 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:07:48.777069 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.777001 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token" (OuterVolumeSpecName: "istio-token") pod "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" (UID: "6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:07:48.875655 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875620 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875655 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875649 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875655 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875658 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvmh2\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-kube-api-access-zvmh2\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875901 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875668 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875901 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875676 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875901 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875683 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:48.875901 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:48.875692 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:07:49.090795 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.090696 2549 generic.go:356] "Generic (PLEG): container finished" podID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerID="b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b" exitCode=0 Jul 22 03:07:49.090795 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.090760 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" Jul 22 03:07:49.091023 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.090783 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerDied","Data":"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b"} Jul 22 03:07:49.091023 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.090863 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2" event={"ID":"6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2","Type":"ContainerDied","Data":"17210857b11954a7408075f3725068cbbe05b5e6381f6e47b0a5487829a5c44c"} Jul 22 03:07:49.091023 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.090895 2549 scope.go:117] "RemoveContainer" containerID="b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b" Jul 22 03:07:49.100679 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.100663 2549 scope.go:117] "RemoveContainer" containerID="1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90" Jul 22 03:07:49.108523 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.108506 2549 scope.go:117] "RemoveContainer" containerID="b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b" Jul 22 03:07:49.108752 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:07:49.108737 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b\": container with ID starting with b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b not found: ID does not exist" containerID="b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b" Jul 22 03:07:49.108791 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.108760 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b"} err="failed to get container status \"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b\": rpc error: code = NotFound desc = could not find container \"b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b\": container with ID starting with b52c36344905108d03faa413045468e9a705912cf4b7767ae7397f548969f08b not found: ID does not exist" Jul 22 03:07:49.108791 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.108778 2549 scope.go:117] "RemoveContainer" containerID="1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90" Jul 22 03:07:49.108980 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:07:49.108966 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90\": container with ID starting with 1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90 not found: ID does not exist" containerID="1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90" Jul 22 03:07:49.109019 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.108984 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90"} err="failed to get container status \"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90\": rpc error: code = NotFound desc = could not find container \"1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90\": container with ID starting with 1cc16f54672bf2fd4a100fbd3b1af33e704de4635b0a43e07874818ca4ff4c90 not found: ID does not exist" Jul 22 03:07:49.113291 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.113273 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:07:49.118088 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.118064 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f3887a-openshift-default-77848b65cf-csfm2"] Jul 22 03:07:49.379245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:49.379165 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" path="/var/lib/kubelet/pods/6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2/volumes" Jul 22 03:07:51.958946 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.958913 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:07:51.959304 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959269 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.959304 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959280 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.959304 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959294 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.959304 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959301 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.959499 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959397 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.959499 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.959438 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5dffc5-595b-4075-a1c7-ba2e9f58e9d2" containerName="istio-proxy" Jul 22 03:07:51.963871 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.963853 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:51.971576 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.971558 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-e6d029-openshift-default-dockercfg-fttkq\"" Jul 22 03:07:51.994512 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:51.994484 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:07:52.002553 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002527 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002669 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002574 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002669 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002630 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002774 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002678 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002774 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002704 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002851 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002790 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv8q6\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002851 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002833 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002929 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002880 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.002929 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.002920 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104153 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104117 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv8q6\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104153 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104159 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104378 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104194 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104378 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104222 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104378 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104299 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104590 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104479 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104652 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104598 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104708 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104659 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104708 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104685 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104708 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104684 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.104875 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.104728 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.105041 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.105019 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.105162 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.105022 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.105204 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.105104 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.106728 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.106704 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.106868 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.106851 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.114257 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.114234 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.114492 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.114446 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv8q6\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6\") pod \"e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.274218 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.274119 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:52.411327 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.411303 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:07:52.413136 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:07:52.413115 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61f2846_48d6_4e47_82cf_d7904fafe086.slice/crio-fac9f5da2f8eef22cc658ee98de8f7ab7d858ef83852e527d59b4cbedfafa614 WatchSource:0}: Error finding container fac9f5da2f8eef22cc658ee98de8f7ab7d858ef83852e527d59b4cbedfafa614: Status 404 returned error can't find the container with id fac9f5da2f8eef22cc658ee98de8f7ab7d858ef83852e527d59b4cbedfafa614 Jul 22 03:07:52.415658 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.415630 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:52.415732 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.415694 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:52.415773 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:52.415743 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:07:53.119783 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:53.119749 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerStarted","Data":"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378"} Jul 22 03:07:53.120233 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:53.119802 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerStarted","Data":"fac9f5da2f8eef22cc658ee98de8f7ab7d858ef83852e527d59b4cbedfafa614"} Jul 22 03:07:53.141841 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:53.141776 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podStartSLOduration=2.141762326 podStartE2EDuration="2.141762326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:53.139050948 +0000 UTC m=+2364.296063231" watchObservedRunningTime="2026-07-22 03:07:53.141762326 +0000 UTC m=+2364.298774608" Jul 22 03:07:53.275317 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:53.275280 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:53.280288 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:53.280263 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:54.124875 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:54.124844 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:54.125852 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:54.125829 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:07:54.325281 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:07:54.325243 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:08:08.432832 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:08.432796 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:08:10.166596 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:10.166550 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 22 03:08:10.167024 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:10.166635 2549 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 22 03:08:11.216320 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.216289 2549 generic.go:356] "Generic (PLEG): container finished" podID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerID="d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378" exitCode=0 Jul 22 03:08:11.216741 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.216364 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerDied","Data":"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378"} Jul 22 03:08:11.216797 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.216783 2549 scope.go:117] "RemoveContainer" containerID="d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378" Jul 22 03:08:11.217893 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.217867 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:08:11.217967 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.217950 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:08:11.218015 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:11.218002 2549 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 22 03:08:12.224304 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:12.224269 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerStarted","Data":"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9"} Jul 22 03:08:12.274526 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:12.274479 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:08:12.275029 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:12.274999 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 03:08:12.275254 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:12.275226 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 03:08:13.275541 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:13.275497 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 03:08:13.275928 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:13.275576 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 03:08:14.275157 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:14.275114 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 03:08:14.275371 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:14.275178 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 03:08:15.275069 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:15.275034 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 03:08:15.275464 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:15.275086 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 03:08:16.274754 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:16.274716 2549 patch_prober.go:28] interesting pod/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 22 03:08:16.274930 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:16.274769 2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 22 03:08:17.277810 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:17.277778 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:08:17.278262 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:17.278149 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:08:17.278902 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:17.278880 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:08:18.233771 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:18.233732 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:08:29.587878 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:29.587846 2549 scope.go:117] "RemoveContainer" containerID="a25a7da06756ac54f6f74bae8a41ab6f6488fcac437e98c5e1e3297d1aad86ba" Jul 22 03:08:34.623514 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:08:34.623472 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:09:04.024700 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:09:04.024616 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:09:19.654625 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:09:19.654586 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:09:36.924149 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:09:36.924104 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:09:54.034362 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:09:54.034322 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:10:11.729731 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:11.729690 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:10:28.134784 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:28.134703 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:10:42.595404 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.595371 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:10:42.598315 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.598295 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.601846 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.601825 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"kube-root-ca.crt\"" Jul 22 03:10:42.601960 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.601858 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"sub-test-model-643135-kserve-self-signed-certs\"" Jul 22 03:10:42.601960 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.601863 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"default-dockercfg-5w4zg\"" Jul 22 03:10:42.602112 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.602096 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"openshift-service-ca.crt\"" Jul 22 03:10:42.610672 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.610652 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:10:42.741009 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.740967 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.741208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.741040 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxsrl\" (UniqueName: \"kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.741208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.741074 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.741208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.741090 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.741208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.741166 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.741208 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.741187 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842183 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842147 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842183 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842182 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842619 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842332 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842619 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842475 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxsrl\" (UniqueName: \"kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842619 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842532 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842757 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842628 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842757 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842637 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842757 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842676 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.842915 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.842799 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.844682 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.844655 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.845160 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.845143 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.851564 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.851501 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxsrl\" (UniqueName: \"kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl\") pod \"sub-test-model-643135-kserve-59b5798f87-6w5l8\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:42.907265 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:42.907224 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:43.048764 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:43.048735 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:10:43.050080 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:10:43.050056 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ca1afdc_02c6_4658_9931_38cbad3c9adb.slice/crio-b3106c320f99b666599795485d8fa7553067f14e7be8dc50d301846ad514f0c4 WatchSource:0}: Error finding container b3106c320f99b666599795485d8fa7553067f14e7be8dc50d301846ad514f0c4: Status 404 returned error can't find the container with id b3106c320f99b666599795485d8fa7553067f14e7be8dc50d301846ad514f0c4 Jul 22 03:10:44.026547 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:44.026513 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" event={"ID":"1ca1afdc-02c6-4658-9931-38cbad3c9adb","Type":"ContainerStarted","Data":"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c"} Jul 22 03:10:44.026963 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:44.026555 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" event={"ID":"1ca1afdc-02c6-4658-9931-38cbad3c9adb","Type":"ContainerStarted","Data":"b3106c320f99b666599795485d8fa7553067f14e7be8dc50d301846ad514f0c4"} Jul 22 03:10:44.048523 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:44.048476 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" podStartSLOduration=2.048462396 podStartE2EDuration="2.048462396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:10:44.046874939 +0000 UTC m=+2535.203887222" watchObservedRunningTime="2026-07-22 03:10:44.048462396 +0000 UTC m=+2535.205474679" Jul 22 03:10:46.834757 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:46.834718 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:10:52.907760 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:52.907698 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:52.907760 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:52.907774 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:52.920338 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:52.920313 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:10:53.082516 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:10:53.082481 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:11:05.125617 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:05.125579 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:11:19.128725 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:19.128684 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:11:38.629135 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:38.629096 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:11:44.780782 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:44.780743 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:11:56.228313 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:56.228274 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:11:59.802297 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:11:59.802233 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" podUID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" containerName="main" containerID="cri-o://6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c" gracePeriod=45 Jul 22 03:12:00.067653 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.067623 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:12:00.145712 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145668 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxsrl\" (UniqueName: \"kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.145900 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145739 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.145900 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145771 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.145900 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145800 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.145900 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145851 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.145900 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.145865 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm\") pod \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\" (UID: \"1ca1afdc-02c6-4658-9931-38cbad3c9adb\") " Jul 22 03:12:00.146479 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.146030 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:00.146479 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.146163 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:00.146479 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.146174 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home" (OuterVolumeSpecName: "home") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:00.148121 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.148093 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm" (OuterVolumeSpecName: "dshm") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:12:00.148121 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.148097 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl" (OuterVolumeSpecName: "kube-api-access-rxsrl") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "kube-api-access-rxsrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:12:00.148263 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.148142 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ca1afdc-02c6-4658-9931-38cbad3c9adb" (UID: "1ca1afdc-02c6-4658-9931-38cbad3c9adb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:12:00.247264 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247225 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxsrl\" (UniqueName: \"kubernetes.io/projected/1ca1afdc-02c6-4658-9931-38cbad3c9adb-kube-api-access-rxsrl\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.247264 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247259 2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.247264 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247270 2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ca1afdc-02c6-4658-9931-38cbad3c9adb-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.247564 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247279 2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.247564 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247287 2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.247564 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.247294 2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ca1afdc-02c6-4658-9931-38cbad3c9adb-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:12:00.415865 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.415764 2549 generic.go:356] "Generic (PLEG): container finished" podID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" containerID="6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c" exitCode=0 Jul 22 03:12:00.415865 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.415812 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" Jul 22 03:12:00.416059 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.415856 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" event={"ID":"1ca1afdc-02c6-4658-9931-38cbad3c9adb","Type":"ContainerDied","Data":"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c"} Jul 22 03:12:00.416059 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.415901 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8" event={"ID":"1ca1afdc-02c6-4658-9931-38cbad3c9adb","Type":"ContainerDied","Data":"b3106c320f99b666599795485d8fa7553067f14e7be8dc50d301846ad514f0c4"} Jul 22 03:12:00.416059 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.415920 2549 scope.go:117] "RemoveContainer" containerID="6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c" Jul 22 03:12:00.425499 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.425475 2549 scope.go:117] "RemoveContainer" containerID="6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c" Jul 22 03:12:00.425809 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:12:00.425791 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c\": container with ID starting with 6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c not found: ID does not exist" containerID="6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c" Jul 22 03:12:00.425851 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.425822 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c"} err="failed to get container status \"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c\": rpc error: code = NotFound desc = could not find container \"6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c\": container with ID starting with 6260a4f4e971a0bb58db856cd81cce2cc0910b1cb3f51f2f088d841e8bea475c not found: ID does not exist" Jul 22 03:12:00.438085 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.438053 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:12:00.441792 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:00.441760 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-643135/sub-test-model-643135-kserve-59b5798f87-6w5l8"] Jul 22 03:12:01.380345 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:01.380307 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" path="/var/lib/kubelet/pods/1ca1afdc-02c6-4658-9931-38cbad3c9adb/volumes" Jul 22 03:12:05.424245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:05.424209 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:12:29.635166 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:29.635138 2549 scope.go:117] "RemoveContainer" containerID="3d28d0c5a2102ceabad074cb0a884ab9e78332053756290bbe06f0c97b1208d1" Jul 22 03:12:31.310721 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.310687 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:12:31.311118 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.311031 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" containerName="main" Jul 22 03:12:31.311118 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.311041 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" containerName="main" Jul 22 03:12:31.311118 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.311101 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ca1afdc-02c6-4658-9931-38cbad3c9adb" containerName="main" Jul 22 03:12:31.315549 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.315529 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.318257 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.318223 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"openshift-service-ca.crt\"" Jul 22 03:12:31.319339 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.319322 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"rate-test-model-e6d029-kserve-self-signed-certs\"" Jul 22 03:12:31.319475 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.319324 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"default-dockercfg-x8dhn\"" Jul 22 03:12:31.319475 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.319356 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"kube-root-ca.crt\"" Jul 22 03:12:31.324594 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.324571 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:12:31.327320 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327299 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.327439 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327329 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.327439 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327352 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzfpk\" (UniqueName: \"kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.327554 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327438 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.327554 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327480 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.327667 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.327578 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428724 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428684 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428925 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428742 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428925 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428771 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428925 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428817 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428925 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428836 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.428925 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.428857 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzfpk\" (UniqueName: \"kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.429291 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.429263 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.429381 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.429289 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.429381 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.429343 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.431166 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.431143 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.431402 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.431383 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.439330 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.439307 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzfpk\" (UniqueName: \"kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk\") pod \"rate-test-model-e6d029-kserve-64cd4769f5-bkstv\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.626261 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.626158 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:31.768017 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:31.767987 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:12:31.769915 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:12:31.769893 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeca35dd2_4b24_4b97_8bdc_7eadde26253d.slice/crio-153b108d8e7b14ca793152a67b78e0eeacc1aecdb4be54be41325994e8703862 WatchSource:0}: Error finding container 153b108d8e7b14ca793152a67b78e0eeacc1aecdb4be54be41325994e8703862: Status 404 returned error can't find the container with id 153b108d8e7b14ca793152a67b78e0eeacc1aecdb4be54be41325994e8703862 Jul 22 03:12:32.580967 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:32.580929 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" event={"ID":"eca35dd2-4b24-4b97-8bdc-7eadde26253d","Type":"ContainerStarted","Data":"b40600aa3324d9f7a81ce889c886594f7620aa60a4c2b8484461608612dc8a69"} Jul 22 03:12:32.581400 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:32.580978 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" event={"ID":"eca35dd2-4b24-4b97-8bdc-7eadde26253d","Type":"ContainerStarted","Data":"153b108d8e7b14ca793152a67b78e0eeacc1aecdb4be54be41325994e8703862"} Jul 22 03:12:32.600212 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:32.600154 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" podStartSLOduration=1.60013957 podStartE2EDuration="1.60013957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:12:32.597947563 +0000 UTC m=+2643.754959847" watchObservedRunningTime="2026-07-22 03:12:32.60013957 +0000 UTC m=+2643.757151853" Jul 22 03:12:35.031853 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:35.031809 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:12:41.626433 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:41.626386 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:41.626854 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:41.626451 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:41.639524 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:41.639490 2549 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:41.650609 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:41.650591 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:12:49.070937 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:12:49.070889 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:13:02.534162 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:02.534124 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:13:08.693406 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:08.693366 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:13:19.633322 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:19.633233 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:13:23.714326 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:23.714270 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" podUID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" containerName="main" containerID="cri-o://b40600aa3324d9f7a81ce889c886594f7620aa60a4c2b8484461608612dc8a69" gracePeriod=45 Jul 22 03:13:23.866054 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:23.866018 2549 generic.go:356] "Generic (PLEG): container finished" podID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" containerID="b40600aa3324d9f7a81ce889c886594f7620aa60a4c2b8484461608612dc8a69" exitCode=0 Jul 22 03:13:23.866234 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:23.866088 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" event={"ID":"eca35dd2-4b24-4b97-8bdc-7eadde26253d","Type":"ContainerDied","Data":"b40600aa3324d9f7a81ce889c886594f7620aa60a4c2b8484461608612dc8a69"} Jul 22 03:13:23.960514 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:23.960489 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:13:24.005023 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.004939 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005023 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005013 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005065 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005116 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzfpk\" (UniqueName: \"kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005152 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005185 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache\") pod \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\" (UID: \"eca35dd2-4b24-4b97-8bdc-7eadde26253d\") " Jul 22 03:13:24.005245 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005202 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:24.005526 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005269 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home" (OuterVolumeSpecName: "home") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:24.005526 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005462 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache" (OuterVolumeSpecName: "model-cache") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:24.005609 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005577 2549 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-model-cache\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.005609 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005597 2549 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tmp-dir\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.005687 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.005609 2549 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-home\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.007576 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.007551 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk" (OuterVolumeSpecName: "kube-api-access-pzfpk") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "kube-api-access-pzfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:13:24.007826 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.007798 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 03:13:24.007826 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.007812 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm" (OuterVolumeSpecName: "dshm") pod "eca35dd2-4b24-4b97-8bdc-7eadde26253d" (UID: "eca35dd2-4b24-4b97-8bdc-7eadde26253d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:13:24.106204 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.106171 2549 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca35dd2-4b24-4b97-8bdc-7eadde26253d-tls-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.106204 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.106202 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzfpk\" (UniqueName: \"kubernetes.io/projected/eca35dd2-4b24-4b97-8bdc-7eadde26253d-kube-api-access-pzfpk\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.106406 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.106212 2549 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca35dd2-4b24-4b97-8bdc-7eadde26253d-dshm\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:24.261428 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.261327 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7912-140717351254032"] Jul 22 03:13:24.261699 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.261686 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" containerName="main" Jul 22 03:13:24.261751 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.261701 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" containerName="main" Jul 22 03:13:24.261790 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.261771 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" containerName="main" Jul 22 03:13:24.264933 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.264916 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:24.267668 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.267642 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vnmk7\"" Jul 22 03:13:24.270740 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.270716 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"] Jul 22 03:13:24.307810 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.307781 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljxn\" (UniqueName: \"kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn\") pod \"test-curl-7912-140717351254032\" (UID: \"8edd2dea-f700-4710-9a4f-1cbf2e95396b\") " pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:24.408340 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.408303 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljxn\" (UniqueName: \"kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn\") pod \"test-curl-7912-140717351254032\" (UID: \"8edd2dea-f700-4710-9a4f-1cbf2e95396b\") " pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:24.416745 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.416720 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljxn\" (UniqueName: \"kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn\") pod \"test-curl-7912-140717351254032\" (UID: \"8edd2dea-f700-4710-9a4f-1cbf2e95396b\") " pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:24.575524 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.575434 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:24.708917 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.708888 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"] Jul 22 03:13:24.711330 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:13:24.711301 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8edd2dea_f700_4710_9a4f_1cbf2e95396b.slice/crio-589a1279a64ecef807d4db2848f78bee473d066f35a59990664c4f104f0ddbf7 WatchSource:0}: Error finding container 589a1279a64ecef807d4db2848f78bee473d066f35a59990664c4f104f0ddbf7: Status 404 returned error can't find the container with id 589a1279a64ecef807d4db2848f78bee473d066f35a59990664c4f104f0ddbf7 Jul 22 03:13:24.712577 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.712560 2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 03:13:24.872540 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.872436 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351254032" event={"ID":"8edd2dea-f700-4710-9a4f-1cbf2e95396b","Type":"ContainerStarted","Data":"589a1279a64ecef807d4db2848f78bee473d066f35a59990664c4f104f0ddbf7"} Jul 22 03:13:24.873766 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.873738 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" event={"ID":"eca35dd2-4b24-4b97-8bdc-7eadde26253d","Type":"ContainerDied","Data":"153b108d8e7b14ca793152a67b78e0eeacc1aecdb4be54be41325994e8703862"} Jul 22 03:13:24.873921 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.873783 2549 scope.go:117] "RemoveContainer" containerID="b40600aa3324d9f7a81ce889c886594f7620aa60a4c2b8484461608612dc8a69" Jul 22 03:13:24.873921 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.873791 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv" Jul 22 03:13:24.896787 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.896761 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:13:24.900353 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:24.900330 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/rate-test-model-e6d029-kserve-64cd4769f5-bkstv"] Jul 22 03:13:25.382251 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:25.382205 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca35dd2-4b24-4b97-8bdc-7eadde26253d" path="/var/lib/kubelet/pods/eca35dd2-4b24-4b97-8bdc-7eadde26253d/volumes" Jul 22 03:13:25.879957 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:25.879925 2549 generic.go:356] "Generic (PLEG): container finished" podID="8edd2dea-f700-4710-9a4f-1cbf2e95396b" containerID="67d9e66df50508ecd45cc72a54d68d6646a6faeede30bc2d3e9643b1dd74f06d" exitCode=0 Jul 22 03:13:25.880361 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:25.880017 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351254032" event={"ID":"8edd2dea-f700-4710-9a4f-1cbf2e95396b","Type":"ContainerDied","Data":"67d9e66df50508ecd45cc72a54d68d6646a6faeede30bc2d3e9643b1dd74f06d"} Jul 22 03:13:27.023047 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.023016 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:27.059738 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.059709 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7912-140717351254032_8edd2dea-f700-4710-9a4f-1cbf2e95396b/test-curl-7912-140717351254032/0.log" Jul 22 03:13:27.085545 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.085511 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"] Jul 22 03:13:27.087575 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.087550 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"] Jul 22 03:13:27.132947 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.132915 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ljxn\" (UniqueName: \"kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn\") pod \"8edd2dea-f700-4710-9a4f-1cbf2e95396b\" (UID: \"8edd2dea-f700-4710-9a4f-1cbf2e95396b\") " Jul 22 03:13:27.135215 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.135191 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn" (OuterVolumeSpecName: "kube-api-access-6ljxn") pod "8edd2dea-f700-4710-9a4f-1cbf2e95396b" (UID: "8edd2dea-f700-4710-9a4f-1cbf2e95396b"). InnerVolumeSpecName "kube-api-access-6ljxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:13:27.234256 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.234221 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ljxn\" (UniqueName: \"kubernetes.io/projected/8edd2dea-f700-4710-9a4f-1cbf2e95396b-kube-api-access-6ljxn\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:13:27.379317 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.379273 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8edd2dea-f700-4710-9a4f-1cbf2e95396b" path="/var/lib/kubelet/pods/8edd2dea-f700-4710-9a4f-1cbf2e95396b/volumes" Jul 22 03:13:27.892931 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.892906 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032" Jul 22 03:13:27.893111 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:27.892950 2549 scope.go:117] "RemoveContainer" containerID="67d9e66df50508ecd45cc72a54d68d6646a6faeede30bc2d3e9643b1dd74f06d" Jul 22 03:13:40.830294 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:13:40.830248 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:14:03.143893 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:03.143853 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-z2rxn"] Jul 22 03:14:11.280553 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:11.280520 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:14:11.280988 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:11.280746 2549 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" containerID="cri-o://d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9" gracePeriod=30 Jul 22 03:14:16.535842 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.535812 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:14:16.574092 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574025 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574092 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574065 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574279 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574120 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574279 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574136 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574279 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574217 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574279 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574268 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574507 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574310 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574507 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574376 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574507 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574402 2549 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vv8q6\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6\") pod \"d61f2846-48d6-4e47-82cf-d7904fafe086\" (UID: \"d61f2846-48d6-4e47-82cf-d7904fafe086\") " Jul 22 03:14:16.574644 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574551 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 03:14:16.574644 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574560 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:14:16.574644 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574586 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:14:16.574789 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574704 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:14:16.574847 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574816 2549 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-credential-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.574847 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574834 2549 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d61f2846-48d6-4e47-82cf-d7904fafe086-istiod-ca-cert\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.575117 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574847 2549 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-certs\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.575117 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.574858 2549 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-workload-socket\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.575748 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.575717 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data" (OuterVolumeSpecName: "istio-data") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:14:16.576554 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.576524 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 03:14:16.576774 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.576751 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6" (OuterVolumeSpecName: "kube-api-access-vv8q6") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "kube-api-access-vv8q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:14:16.576854 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.576837 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 03:14:16.576994 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.576973 2549 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token" (OuterVolumeSpecName: "istio-token") pod "d61f2846-48d6-4e47-82cf-d7904fafe086" (UID: "d61f2846-48d6-4e47-82cf-d7904fafe086"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 03:14:16.675976 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.675945 2549 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-podinfo\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.675976 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.675971 2549 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-envoy\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.675976 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.675979 2549 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-data\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.676200 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.675990 2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv8q6\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-kube-api-access-vv8q6\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:16.676200 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:16.675999 2549 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d61f2846-48d6-4e47-82cf-d7904fafe086-istio-token\") on node \"ip-10-0-133-125.ec2.internal\" DevicePath \"\"" Jul 22 03:14:17.151924 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.151889 2549 generic.go:356] "Generic (PLEG): container finished" podID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerID="d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9" exitCode=0 Jul 22 03:14:17.152080 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.151936 2549 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" Jul 22 03:14:17.152080 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.151979 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerDied","Data":"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9"} Jul 22 03:14:17.152080 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.152031 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw" event={"ID":"d61f2846-48d6-4e47-82cf-d7904fafe086","Type":"ContainerDied","Data":"fac9f5da2f8eef22cc658ee98de8f7ab7d858ef83852e527d59b4cbedfafa614"} Jul 22 03:14:17.152080 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.152054 2549 scope.go:117] "RemoveContainer" containerID="d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9" Jul 22 03:14:17.166922 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.166900 2549 scope.go:117] "RemoveContainer" containerID="d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378" Jul 22 03:14:17.175352 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.175332 2549 scope.go:117] "RemoveContainer" containerID="d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9" Jul 22 03:14:17.175638 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:14:17.175619 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9\": container with ID starting with d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9 not found: ID does not exist" containerID="d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9" Jul 22 03:14:17.175682 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.175646 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9"} err="failed to get container status \"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9\": rpc error: code = NotFound desc = could not find container \"d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9\": container with ID starting with d33155cd22ca8c029b648c5a3e9732b5c0331dc75e9fa638116039b2d73669d9 not found: ID does not exist" Jul 22 03:14:17.175682 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.175662 2549 scope.go:117] "RemoveContainer" containerID="d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378" Jul 22 03:14:17.175893 ip-10-0-133-125 kubenswrapper[2549]: E0722 03:14:17.175875 2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378\": container with ID starting with d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378 not found: ID does not exist" containerID="d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378" Jul 22 03:14:17.175932 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.175899 2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378"} err="failed to get container status \"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378\": rpc error: code = NotFound desc = could not find container \"d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378\": container with ID starting with d92dbb95e093c9f942e87f33804948fe9a9c94aa1abb5d956ee1abd576162378 not found: ID does not exist" Jul 22 03:14:17.181119 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.181096 2549 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:14:17.183204 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.183179 2549 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-e6d029-openshift-default-7bb78499-sd7lw"] Jul 22 03:14:17.380081 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.380046 2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" path="/var/lib/kubelet/pods/d61f2846-48d6-4e47-82cf-d7904fafe086/volumes" Jul 22 03:14:17.783162 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:17.783135 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6765967b89-76trs_d8db8daf-0355-47c0-95fb-0ecdaf1cfe19/manager/0.log" Jul 22 03:14:18.295559 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:18.295527 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-phdrf_65435ad2-7f85-4955-ac4f-b2954847459e/manager/0.log" Jul 22 03:14:18.301852 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:18.301832 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-phdrf_65435ad2-7f85-4955-ac4f-b2954847459e/copy-manifests/0.log" Jul 22 03:14:19.004913 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:19.004884 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744820-x8hz7_3aa08604-39eb-48e4-b00e-b129ebda87e2/cleanup/0.log" Jul 22 03:14:19.930834 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:19.930804 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/util/0.log" Jul 22 03:14:19.938160 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:19.938140 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/pull/0.log" Jul 22 03:14:19.944495 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:19.944479 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/extract/0.log" Jul 22 03:14:20.068783 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.068753 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/util/0.log" Jul 22 03:14:20.075686 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.075663 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/pull/0.log" Jul 22 03:14:20.082960 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.082942 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/extract/0.log" Jul 22 03:14:20.202613 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.202535 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/util/0.log" Jul 22 03:14:20.209614 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.209589 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/pull/0.log" Jul 22 03:14:20.218813 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.218790 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/extract/0.log" Jul 22 03:14:20.331929 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.331905 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/util/0.log" Jul 22 03:14:20.338904 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.338884 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/pull/0.log" Jul 22 03:14:20.345339 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:20.345321 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/extract/0.log" Jul 22 03:14:21.218640 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:21.218609 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-z2rxn_a7348d98-514b-4875-a12b-de1d9cd807c5/limitador/0.log" Jul 22 03:14:21.704860 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:21.704831 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j8wkt_32838d19-072b-4b8b-86cd-ff53120ed979/istio-proxy/0.log" Jul 22 03:14:21.823024 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:21.822993 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7kqgx_bf3ff7db-5356-4305-ab03-1f6892c1a38b/discovery/0.log" Jul 22 03:14:23.260558 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:23.260523 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp45s2_687a7e41-d64b-40ce-bc5a-9e14783044a4/main/0.log" Jul 22 03:14:32.436061 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:32.436025 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q6k5b_2a35637e-f5c1-452f-9b43-46f19f25314f/global-pull-secret-syncer/0.log" Jul 22 03:14:32.597631 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:32.597597 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r45jl_abf512bb-898b-4547-8286-6219249d80f9/konnectivity-agent/0.log" Jul 22 03:14:32.644452 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:32.644400 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-125.ec2.internal_5786553615a694db0fe10013f4d054ba/haproxy/0.log" Jul 22 03:14:37.094431 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.094389 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/extract/0.log" Jul 22 03:14:37.127679 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.127651 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/util/0.log" Jul 22 03:14:37.154788 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.154758 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cwf8f_1b4a216d-c4a3-499d-b410-b055eef44fc8/pull/0.log" Jul 22 03:14:37.186707 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.186659 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/extract/0.log" Jul 22 03:14:37.214562 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.214529 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/util/0.log" Jul 22 03:14:37.239567 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.239542 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qt9rd_456c911f-b427-4e93-a6c8-09bdbbdce24b/pull/0.log" Jul 22 03:14:37.269393 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.269369 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/extract/0.log" Jul 22 03:14:37.294180 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.294155 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/util/0.log" Jul 22 03:14:37.322543 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.322520 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fpbtz_acc4d111-f49f-47b7-9c02-94523f4f880b/pull/0.log" Jul 22 03:14:37.356643 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.356585 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/extract/0.log" Jul 22 03:14:37.396162 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.396135 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/util/0.log" Jul 22 03:14:37.418590 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:37.418567 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1n6s4t_653d7521-ee37-4a47-b0ec-d8d52c53a971/pull/0.log" Jul 22 03:14:38.338910 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:38.338878 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-z2rxn_a7348d98-514b-4875-a12b-de1d9cd807c5/limitador/0.log" Jul 22 03:14:40.444632 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.444598 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-r9972_3f856281-f75d-4bad-af78-4a0fbc449557/kube-state-metrics/0.log" Jul 22 03:14:40.468826 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.468799 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-r9972_3f856281-f75d-4bad-af78-4a0fbc449557/kube-rbac-proxy-main/0.log" Jul 22 03:14:40.495548 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.495515 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-r9972_3f856281-f75d-4bad-af78-4a0fbc449557/kube-rbac-proxy-self/0.log" Jul 22 03:14:40.534545 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.534515 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7657bfd955-8vx2n_eb6480fc-238b-4942-b5f3-6d6b9ad80828/metrics-server/0.log" Jul 22 03:14:40.789971 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.789900 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7dqn_6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b/node-exporter/0.log" Jul 22 03:14:40.823401 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.823372 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7dqn_6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b/kube-rbac-proxy/0.log" Jul 22 03:14:40.851762 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.851737 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m7dqn_6d8b6f0b-04f8-45d4-a13c-d1c77fb5843b/init-textfile/0.log" Jul 22 03:14:40.890407 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.890383 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ws4pb_bc1642c6-511d-4287-9919-e5c97f63a516/kube-rbac-proxy-main/0.log" Jul 22 03:14:40.917477 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.917450 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ws4pb_bc1642c6-511d-4287-9919-e5c97f63a516/kube-rbac-proxy-self/0.log" Jul 22 03:14:40.943394 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:40.943370 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-ws4pb_bc1642c6-511d-4287-9919-e5c97f63a516/openshift-state-metrics/0.log" Jul 22 03:14:41.321689 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.321660 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86856444f-skxsq_371141c7-71f4-4f45-95e2-4c623758459c/telemeter-client/0.log" Jul 22 03:14:41.347390 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.347350 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86856444f-skxsq_371141c7-71f4-4f45-95e2-4c623758459c/reload/0.log" Jul 22 03:14:41.377597 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.377565 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-86856444f-skxsq_371141c7-71f4-4f45-95e2-4c623758459c/kube-rbac-proxy/0.log" Jul 22 03:14:41.417546 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.417515 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/thanos-query/0.log" Jul 22 03:14:41.442224 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.442197 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/kube-rbac-proxy-web/0.log" Jul 22 03:14:41.466382 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.466356 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/kube-rbac-proxy/0.log" Jul 22 03:14:41.491321 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.491298 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/prom-label-proxy/0.log" Jul 22 03:14:41.513780 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.513756 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/kube-rbac-proxy-rules/0.log" Jul 22 03:14:41.537938 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:41.537908 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74c59d48c5-rngbl_a9862596-c307-4d89-a162-2d649da6c210/kube-rbac-proxy-metrics/0.log" Jul 22 03:14:42.782305 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782269 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx"] Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782682 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782699 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782710 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8edd2dea-f700-4710-9a4f-1cbf2e95396b" containerName="test-curl-7912-140717351254032" Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782716 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8edd2dea-f700-4710-9a4f-1cbf2e95396b" containerName="test-curl-7912-140717351254032" Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782729 2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:14:42.782753 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782735 2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:14:42.782982 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782787 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:14:42.782982 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.782794 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="8edd2dea-f700-4710-9a4f-1cbf2e95396b" containerName="test-curl-7912-140717351254032" Jul 22 03:14:42.785837 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.785818 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:42.788458 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.788435 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gt8kb\"/\"openshift-service-ca.crt\"" Jul 22 03:14:42.788580 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.788466 2549 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gt8kb\"/\"kube-root-ca.crt\"" Jul 22 03:14:42.789259 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.789242 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gt8kb\"/\"default-dockercfg-sxttz\"" Jul 22 03:14:42.795566 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.795543 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx"] Jul 22 03:14:42.909203 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.909163 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-lib-modules\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:42.909398 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.909249 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-proc\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:42.909398 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.909268 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-sys\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:42.909398 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.909300 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-podres\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:42.909398 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:42.909322 2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jzf\" (UniqueName: \"kubernetes.io/projected/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-kube-api-access-r2jzf\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010194 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010151 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-podres\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010194 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010198 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jzf\" (UniqueName: \"kubernetes.io/projected/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-kube-api-access-r2jzf\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010496 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010250 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-lib-modules\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010496 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010332 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-podres\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010496 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010443 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-proc\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010496 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010467 2549 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-sys\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010677 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010408 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-lib-modules\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010677 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010551 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-proc\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.010677 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.010558 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-sys\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.019353 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.019329 2549 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jzf\" (UniqueName: \"kubernetes.io/projected/338824e6-64e8-4a5e-8be9-dcc2a11b9f7f-kube-api-access-r2jzf\") pod \"perf-node-gather-daemonset-pqdxx\" (UID: \"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.096433 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.096339 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:43.235594 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.235562 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx"] Jul 22 03:14:43.237449 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:14:43.237406 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod338824e6_64e8_4a5e_8be9_dcc2a11b9f7f.slice/crio-0b032aee48f0af98cb181922d193fea96ccc3fb97aee9655156b99db18618326 WatchSource:0}: Error finding container 0b032aee48f0af98cb181922d193fea96ccc3fb97aee9655156b99db18618326: Status 404 returned error can't find the container with id 0b032aee48f0af98cb181922d193fea96ccc3fb97aee9655156b99db18618326 Jul 22 03:14:43.288402 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:43.288370 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" event={"ID":"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f","Type":"ContainerStarted","Data":"0b032aee48f0af98cb181922d193fea96ccc3fb97aee9655156b99db18618326"} Jul 22 03:14:44.296113 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:44.296071 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" event={"ID":"338824e6-64e8-4a5e-8be9-dcc2a11b9f7f","Type":"ContainerStarted","Data":"f842b14b70b64169e328bb033980b9b8a140dcd3e2cb01db2676c13d1a00972f"} Jul 22 03:14:44.296605 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:44.296133 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:44.314698 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:44.314655 2549 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" podStartSLOduration=2.314640211 podStartE2EDuration="2.314640211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 03:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:14:44.312258573 +0000 UTC m=+2775.469270835" watchObservedRunningTime="2026-07-22 03:14:44.314640211 +0000 UTC m=+2775.471652493" Jul 22 03:14:45.815713 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:45.815681 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqk89_50c672fa-b2ef-473b-8ab5-550546d03174/dns/0.log" Jul 22 03:14:45.889970 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:45.889943 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqk89_50c672fa-b2ef-473b-8ab5-550546d03174/kube-rbac-proxy/0.log" Jul 22 03:14:46.015038 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:46.015003 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qbfxt_50f601a1-dfc7-45a1-9028-0b514b932161/dns-node-resolver/0.log" Jul 22 03:14:46.633450 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:46.633403 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-84dd6fdd8-dp7dt_ea3c4fa3-844c-4bbf-853b-d4740a2d0576/registry/0.log" Jul 22 03:14:46.685498 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:46.685464 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j8qnh_b0a71a19-21a0-4493-b9ae-c8b2c8e9845c/node-ca/0.log" Jul 22 03:14:47.722296 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:47.722255 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j8wkt_32838d19-072b-4b8b-86cd-ff53120ed979/istio-proxy/0.log" Jul 22 03:14:47.850577 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:47.850544 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7kqgx_bf3ff7db-5356-4305-ab03-1f6892c1a38b/discovery/0.log" Jul 22 03:14:48.848683 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:48.848653 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9wdkj_97f5f013-8a28-4018-a9ce-92aaba038aad/serve-healthcheck-canary/0.log" Jul 22 03:14:49.508810 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:49.508735 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpslx_6efbeda9-2838-4cf6-aeef-21a682e27f19/kube-rbac-proxy/0.log" Jul 22 03:14:49.542267 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:49.542235 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpslx_6efbeda9-2838-4cf6-aeef-21a682e27f19/exporter/0.log" Jul 22 03:14:49.565508 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:49.565481 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jpslx_6efbeda9-2838-4cf6-aeef-21a682e27f19/extractor/0.log" Jul 22 03:14:50.314352 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:50.314274 2549 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-pqdxx" Jul 22 03:14:52.242033 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:52.241997 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6765967b89-76trs_d8db8daf-0355-47c0-95fb-0ecdaf1cfe19/manager/0.log" Jul 22 03:14:52.544193 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:52.544167 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-phdrf_65435ad2-7f85-4955-ac4f-b2954847459e/manager/0.log" Jul 22 03:14:52.571811 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:52.571783 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-phdrf_65435ad2-7f85-4955-ac4f-b2954847459e/copy-manifests/0.log" Jul 22 03:14:53.446064 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:53.446030 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j8wkt_32838d19-072b-4b8b-86cd-ff53120ed979/istio-proxy/0.log" Jul 22 03:14:53.566219 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:53.566183 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7kqgx_bf3ff7db-5356-4305-ab03-1f6892c1a38b/discovery/0.log" Jul 22 03:14:54.103988 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:54.103903 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d76bdcd7-hnf5n_714e8fc9-0bc7-415b-8120-e1cd890667ac/manager/0.log" Jul 22 03:14:54.162494 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:54.162459 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-f4tc6_7d8baee0-c70e-40ae-8c83-ad5937999806/openshift-lws-operator/0.log" Jul 22 03:14:54.944372 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:54.944345 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j8wkt_32838d19-072b-4b8b-86cd-ff53120ed979/istio-proxy/0.log" Jul 22 03:14:55.065874 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:14:55.065785 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-7kqgx_bf3ff7db-5356-4305-ab03-1f6892c1a38b/discovery/0.log" Jul 22 03:15:00.144762 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.144717 2549 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr"] Jul 22 03:15:00.145332 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.145310 2549 memory_manager.go:368] "RemoveStaleState removing state" podUID="d61f2846-48d6-4e47-82cf-d7904fafe086" containerName="istio-proxy" Jul 22 03:15:00.154236 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.154213 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr" Jul 22 03:15:00.157499 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.157470 2549 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\"" Jul 22 03:15:00.158276 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.158249 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr"] Jul 22 03:15:00.165590 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.165567 2549 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr" Jul 22 03:15:00.304581 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.304551 2549 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr"] Jul 22 03:15:00.305757 ip-10-0-133-125 kubenswrapper[2549]: W0722 03:15:00.305731 2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40128f32_c1cc_4c6e_bea6_0a71e3b0f5d7.slice/crio-bfc17d6be97b59418a23f7bc9dade906ede3802f1eb6f28690bffbb85df8a3ee WatchSource:0}: Error finding container bfc17d6be97b59418a23f7bc9dade906ede3802f1eb6f28690bffbb85df8a3ee: Status 404 returned error can't find the container with id bfc17d6be97b59418a23f7bc9dade906ede3802f1eb6f28690bffbb85df8a3ee Jul 22 03:15:00.386553 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.386521 2549 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744835-drfkr" event={"ID":"40128f32-c1cc-4c6e-bea6-0a71e3b0f5d7","Type":"ContainerStarted","Data":"bfc17d6be97b59418a23f7bc9dade906ede3802f1eb6f28690bffbb85df8a3ee"} Jul 22 03:15:00.538160 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.538136 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/kube-multus-additional-cni-plugins/0.log" Jul 22 03:15:00.562065 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.562038 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/egress-router-binary-copy/0.log" Jul 22 03:15:00.585820 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.585792 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/cni-plugins/0.log" Jul 22 03:15:00.609385 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.609354 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/bond-cni-plugin/0.log" Jul 22 03:15:00.633405 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.633383 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/routeoverride-cni/0.log" Jul 22 03:15:00.658084 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.658053 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/whereabouts-cni-bincopy/0.log" Jul 22 03:15:00.680050 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:00.680018 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q4cnn_f8641f99-d34b-4f4e-8af0-22b380077196/whereabouts-cni/0.log" Jul 22 03:15:01.135748 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:01.135716 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pp4ls_bdeb088c-7753-438d-a41d-6bf69688c907/kube-multus/0.log" Jul 22 03:15:01.289044 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:01.289000 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lq5wc_d191d66b-70ec-41d9-a66c-dc564775ae3f/network-metrics-daemon/0.log" Jul 22 03:15:01.310966 ip-10-0-133-125 kubenswrapper[2549]: I0722 03:15:01.310935 2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lq5wc_d191d66b-70ec-41d9-a66c-dc564775ae3f/kube-rbac-proxy/0.log"