Jul 16 22:17:16.222149 ip-10-0-141-9 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 22:17:18.198509 ip-10-0-141-9 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:17:18.198509 ip-10-0-141-9 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 22:17:18.198509 ip-10-0-141-9 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:17:18.198509 ip-10-0-141-9 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 22:17:18.198509 ip-10-0-141-9 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 22:17:18.199215 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.198564    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 22:17:18.203828 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203810    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.203828 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203828    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203832    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203836    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203840    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203844    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203847    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203850    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203857    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203860    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203864    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203866    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203869    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203872    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203875    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203878    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203881    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203883    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203886    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203890    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.203890 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203895    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203898    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203902    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203905    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203908    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203911    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203913    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203916    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203919    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203922    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203924    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203927    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203929    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203932    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203934    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203937    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203939    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203942    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203945    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203948    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.204348 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203950    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203953    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203956    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203958    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203961    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203963    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203966    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203968    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203971    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203973    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203976    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203978    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203981    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203984    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203987    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203990    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203993    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203995    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.203998    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204000    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.204869 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204002    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204005    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204007    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204009    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204012    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204015    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204017    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204019    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204022    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204025    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204027    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204030    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204034    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204038    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204043    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204047    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204050    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204055    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204058    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.205349 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204061    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204064    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204067    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204069    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204072    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204074    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204077    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204081    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204084    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204086    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204089    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204092    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204095    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204098    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204101    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204103    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204106    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204108    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204111    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204113    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.205826 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204884    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204892    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204898    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204900    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204903    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204905    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204908    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204911    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204917    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204920    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204922    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204925    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204928    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204931    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204933    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204936    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204939    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204943    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204946    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.206305 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204948    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204951    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204954    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204956    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204959    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204962    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204965    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204967    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204970    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204973    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204975    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204978    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204980    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204983    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204986    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204988    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204990    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204993    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204995    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.204998    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.206808 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205000    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205005    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205010    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205014    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205017    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205019    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205022    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205025    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205028    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205031    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205035    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205038    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205040    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205043    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205045    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205048    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205051    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205054    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205057    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.207300 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205059    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205062    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205064    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205067    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205069    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205071    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205074    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205077    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205079    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205081    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205084    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205087    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205089    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205092    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205094    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205097    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205100    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205103    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205105    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.207783 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205107    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205110    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205112    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205115    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205118    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205121    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205124    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205126    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205131    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205134    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205138    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205142    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205146    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205149    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205152    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205155    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205158    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205160    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205163    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.208283 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205165    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205168    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205547    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205551    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205554    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205557    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205560    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205563    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205566    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205570    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205572    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205576    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205578    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205581    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205583    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205586    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205589    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205591    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205593    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205596    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.208750 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205599    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205601    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205604    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205606    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205608    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205614    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205616    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205619    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205621    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205624    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205627    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205629    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205632    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205634    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205637    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205639    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205646    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205648    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205651    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205654    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.209226 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205656    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205659    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205662    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205664    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205667    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205671    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205674    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205677    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205680    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205682    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205685    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205687    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205690    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205692    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205695    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205697    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205700    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205717    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205720    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205724    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.209749 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205727    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205730    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205732    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205735    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205738    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205741    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205743    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205746    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205750    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205752    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205755    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205757    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205760    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205764    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205768    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205772    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205775    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205778    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205781    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.210238 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205785    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205788    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205791    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205794    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205797    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205800    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205803    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205806    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205808    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205811    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205815    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205818    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205820    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205823    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205825    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205828    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205830    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205833    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205836    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.210716 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205838    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205841    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.205844    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205932    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205942    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205952    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205958    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205962    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205965    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205969    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205974    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205977    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205980    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205984    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205987    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205990    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205993    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205996    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.205999    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206002    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206005    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206008    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206011    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206016    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 22:17:18.211198 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206019    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206023    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206026    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206029    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206033    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206036    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206039    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206042    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206045    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206048    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206051    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206055    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206058    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206062    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206065    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206068    2579 flags.go:64] FLAG: --enable-server="true"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206071    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206077    2579 flags.go:64] FLAG: --event-burst="100"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206080    2579 flags.go:64] FLAG: --event-qps="50"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206084    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206088    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206091    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206095    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206098    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206101    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 22:17:18.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206104    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206107    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206111    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206114    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206117    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206120    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206124    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206127    2579 flags.go:64] FLAG: --feature-gates=""
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206131    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206134    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206137    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206140    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206144    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206147    2579 flags.go:64] FLAG: --help="false"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206150    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206153    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206155    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206158    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206162    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206166    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206169    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206172    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206175    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 22:17:18.212379 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206177    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206180    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206184    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206187    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206190    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206193    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206196    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206199    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206202    2579 flags.go:64] FLAG: --lock-file=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206204    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206207    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206210    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206216    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206218    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206221    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206226    2579 flags.go:64] FLAG: --logging-format="text"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206229    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206232    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206235    2579 flags.go:64] FLAG: --manifest-url=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206237    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206242    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206245    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206254    2579 flags.go:64] FLAG: --max-pods="110"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206257    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206261    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 22:17:18.213090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206264    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206268    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206271    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206273    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206276    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206284    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206286    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206289    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206293    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206296    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206301    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206304    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206308    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206311    2579 flags.go:64] FLAG: --port="10250"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206315    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206317    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d1bb78bff9358c1c"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206321    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206323    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206326    2579 flags.go:64] FLAG: --register-node="true"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206329    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206333    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206335    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206339    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206342    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206384    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 22:17:18.213736 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206389    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206392    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206395    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206398    2579 flags.go:64] FLAG: --runonce="false"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206401    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206404    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206407    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206410    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206413    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206418    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206421    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206425    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206428    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206430    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206433    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206436    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206439    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206442    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206445    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206451    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206454    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206457    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206460    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206463    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206466    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 22:17:18.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206469    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206472    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206475    2579 flags.go:64] FLAG: --v="2"
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206479    2579 flags.go:64] FLAG: --version="false"
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206485    2579 flags.go:64] FLAG: --vmodule=""
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206488    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.206491    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206621    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206626    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206629    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206633    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206636    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206639    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206642    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206645    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206647    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206651    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206654    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206657    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206659    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206662    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.215286 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206665    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206669    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206671    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206674    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206677    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206680    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206682    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206691    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206693    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206696    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206700    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206717    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206720    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206723    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206726    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206730    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206733    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206736    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206739    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206741    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.216030 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206744    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206746    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206749    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206751    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206754    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206757    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206760    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206764    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206767    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206770    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206772    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206775    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206777    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206780    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206783    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206787    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206790    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206793    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206796    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.216561 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206799    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206802    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206806    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206809    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206812    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206815    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206818    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206820    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206825    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206829    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206840    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206843    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206847    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206850    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206852    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206855    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206858    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206860    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206863    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206866    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.217048 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206870    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206872    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206875    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206877    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206880    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206883    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206885    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206888    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206891    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206893    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206896    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206898    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206901    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206903    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206907    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206909    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206912    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206914    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206917    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206920    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.217538 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206924    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206927    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206929    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206932    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206934    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.206937    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.207407    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.216479    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.216498    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.216518    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.216524    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216585    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216591    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216595    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216598    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.218052 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216601    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216604    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216606    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216609    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216612    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216614    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216617    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216620    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216622    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216625    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216627    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216631    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216633    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216636    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216638    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216641    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216644    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216647    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216650    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216653    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.218423 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216656    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216659    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216662    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216665    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216667    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216670    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216673    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216676    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216678    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216681    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216683    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216686    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216689    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216692    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216695    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216697    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216700    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216702    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216723    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216726    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.218921 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216729    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216731    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216735    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216738    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216740    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216743    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216746    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216749    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216751    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216756    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216761    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216764    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216766    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216769    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216772    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216774    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216777    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216780    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216783    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216785    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.219404 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216788    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216791    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216794    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216796    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216799    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216803    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216806    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216809    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216812    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216815    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216818    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216821    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216823    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216826    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216829    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216833    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216836    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216838    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216840    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.219899 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216843    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216846    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216849    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216851    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216854    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216857    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216860    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216862    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216865    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216867    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216870    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216873    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216875    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216878    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216881    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.220368 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.216884    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.216890    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217011    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217016    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217019    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217021    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217024    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217027    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217030    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217032    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217035    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217037    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217042    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217045    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 22:17:18.220756 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217050    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217053    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217056    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217059    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217062    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217065    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217069    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217072    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217074    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217077    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217079    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217082    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217085    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217087    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217090    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217092    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217095    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217097    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217100    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217102    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 22:17:18.221144 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217105    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217108    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217110    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217113    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217116    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217118    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217121    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217123    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217126    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217128    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217131    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217133    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217136    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217140    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217142    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217145    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217148    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217151    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217154    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217157    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 22:17:18.221625 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217159    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217161    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217164    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217167    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217169    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217172    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217174    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217177    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217179    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217182    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217184    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217188    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217192    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217194    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217197    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217200    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217202    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217206    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217209    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 22:17:18.222141 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217212    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217215    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217221    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217224    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217227    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217229    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217233    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217235    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217238    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217244    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217247    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217249    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217252    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217255    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217257    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217260    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217263    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217265    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217268    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 22:17:18.222611 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217271    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217274    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217276    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217279    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217282    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217284    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217287    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:18.217289    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.217295    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.217433    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.220752    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.222306    2579 server.go:1019] "Starting client certificate rotation"
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.222407    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:17:18.223241 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.222440    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 22:17:18.224616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.224601    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:17:18.225829 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.225811    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 22:17:18.235381 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.235358    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 16 22:17:18.237570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.237546    2579 log.go:25] "Validated CRI v1 image API"
Jul 16 22:17:18.238433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.238411    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 22:17:18.241376 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.241347    2579 fs.go:135] Filesystem UUIDs: map[3b310634-f143-41e7-9328-58961e61347f:/dev/nvme0n1p4 5c5cf9c6-abe9-4c75-bdd3-587fe018aa51:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 16 22:17:18.241450 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.241373    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 22:17:18.247659 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.247551    2579 manager.go:216] Machine: {Timestamp:2026-07-16 22:17:18.246395048 +0000 UTC m=+0.314718095 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101140 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2447f3f8f90fdc0e08889b9787d698 SystemUUID:ec2447f3-f8f9-0fdc-0e08-889b9787d698 BootID:0ab276e7-3d87-4002-abc6-d365bbdbd2d8 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b7:2b:95:06:85 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b7:2b:95:06:85 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:a3:7d:36:16:9d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 22:17:18.247659 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.247656    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 22:17:18.247800 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.247758    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 22:17:18.251416 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.251353    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:17:18.251416 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.251356    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 22:17:18.251596 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.251401    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-9.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 22:17:18.252206 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.252195    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 22:17:18.252258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.252209    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 22:17:18.252258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.252217    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:17:18.253001 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.252989    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 22:17:18.254172 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.254161    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 22:17:18.257414 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257397    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:17:18.257549 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257539    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 22:17:18.257607 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257595    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 22:17:18.257607 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257606    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 22:17:18.257670 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257623    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 22:17:18.257670 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257634    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 16 22:17:18.257670 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.257643    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 22:17:18.258062 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.258037    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:17:18.258062 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.258061    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 22:17:18.260550 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.260530    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 22:17:18.262311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.262298    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 22:17:18.262365 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.262328    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 22:17:18.263475 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263465    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 22:17:18.263509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263480    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 22:17:18.263509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263487    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 22:17:18.263509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263492    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 22:17:18.263509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263501    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 22:17:18.263509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263507    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263515    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263520    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263528    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263533    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263541    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263547    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263573    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 22:17:18.263642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.263578    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 22:17:18.266955 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.266942    2579 server.go:1268] "Started kubelet"
Jul 16 22:17:18.267077 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.267045    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 22:17:18.267170 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.267119    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 22:17:18.267215 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.267195    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 22:17:18.268552 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.268527    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-9.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 22:17:18.268956 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.268938    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 22:17:18.269141 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.269095    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 22:17:18.269141 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.269094    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 22:17:18.269487 ip-10-0-141-9 systemd[1]: Started Kubernetes Kubelet.
Jul 16 22:17:18.271237 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.271221    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 22:17:18.274427 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.273509    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-9.ec2.internal.18c2e50b1bdb89eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-9.ec2.internal,UID:ip-10-0-141-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-9.ec2.internal,},FirstTimestamp:2026-07-16 22:17:18.266911211 +0000 UTC m=+0.335234256,LastTimestamp:2026-07-16 22:17:18.266911211 +0000 UTC m=+0.335234256,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-9.ec2.internal,}"
Jul 16 22:17:18.275027 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275011    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 22:17:18.275083 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275034    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 22:17:18.275212 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.275173    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.275702 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275680    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 22:17:18.275826 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275721    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 22:17:18.275826 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275679    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 22:17:18.275916 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275859    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 22:17:18.275916 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.275865    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 22:17:18.276128 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276109    2579 factory.go:55] Registering systemd factory
Jul 16 22:17:18.276188 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276132    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 16 22:17:18.276438 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276421    2579 factory.go:153] Registering CRI-O factory
Jul 16 22:17:18.276438 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276440    2579 factory.go:223] Registration of the crio container factory successfully
Jul 16 22:17:18.276626 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276490    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 22:17:18.276626 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276518    2579 factory.go:103] Registering Raw factory
Jul 16 22:17:18.276626 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.276534    2579 manager.go:1196] Started watching for new ooms in manager
Jul 16 22:17:18.276940 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.276914    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 22:17:18.277274 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.277262    2579 manager.go:349] Starting recovery of all containers
Jul 16 22:17:18.280946 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.280912    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-9.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 22:17:18.281878 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.281657    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 22:17:18.283409 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.283379    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7z897"
Jul 16 22:17:18.287209 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.287168    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 22:17:18.290327 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.290274    2579 manager.go:354] Recovery completed
Jul 16 22:17:18.291893 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.291868    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7z897"
Jul 16 22:17:18.295270 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.295258    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.297286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297136    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.297351 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297300    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.297351 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297310    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.297848 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297831    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 22:17:18.297848 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297842    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.297848 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297846    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 22:17:18.298042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.297865    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 22:17:18.298592 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298575    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.298643 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298608    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.298643 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298619    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.298839 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298827    2579 policy_none.go:49] "None policy: Start"
Jul 16 22:17:18.298877 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298845    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 22:17:18.298877 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.298854    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 22:17:18.299338 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.299234    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-9.ec2.internal.18c2e50b1dab03ce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-9.ec2.internal,UID:ip-10-0-141-9.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-9.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-9.ec2.internal,},FirstTimestamp:2026-07-16 22:17:18.297285582 +0000 UTC m=+0.365608626,LastTimestamp:2026-07-16 22:17:18.297285582 +0000 UTC m=+0.365608626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-9.ec2.internal,}"
Jul 16 22:17:18.299861 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.299846    2579 policy_none.go:47] "Start"
Jul 16 22:17:18.340367 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340346    2579 manager.go:337] "Starting Device Plugin manager"
Jul 16 22:17:18.340505 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.340396    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 22:17:18.340505 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340409    2579 server.go:87] "Starting device plugin registration server"
Jul 16 22:17:18.340744 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340731    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 22:17:18.340791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340746    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 22:17:18.340897 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340877    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 22:17:18.340981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340958    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 22:17:18.340981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.340972    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 22:17:18.341525 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.341497    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 22:17:18.341606 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.341562    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.441189 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.441150    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.442291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.442241    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.442291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.442281    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.442423 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.442295    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.442423 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.442333    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.452843 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.452810    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.452843 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.452843    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-9.ec2.internal\": node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.455647 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.455621    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 22:17:18.455800 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.455664    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 22:17:18.455800 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.455690    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 22:17:18.455800 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.455737    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 22:17:18.457847 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.457827    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:18.496057 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.496024    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.556749 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.556612    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"]
Jul 16 22:17:18.556928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.556768    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.565162 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.565142    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.565266 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.565181    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.565266 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.565190    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.566428 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.566413    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.566525 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.566504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.566570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.566535    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.567326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567305    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.567434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567336    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.567434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567350    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.567434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567350    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.567434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567369    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.567434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.567381    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.568480 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.568463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.568583 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.568496    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 22:17:18.569369 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.569353    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 22:17:18.569463 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.569381    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 22:17:18.569463 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.569394    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeHasSufficientPID"
Jul 16 22:17:18.578212 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.578192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0b61d43fdb33aaaa624fc245668ccfb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"f0b61d43fdb33aaaa624fc245668ccfb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.578290 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.578220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.578290 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.578239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.594605 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.594584    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-9.ec2.internal\" not found" node="ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.596571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.596551    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.599319 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.599299    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-9.ec2.internal\" not found" node="ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679073 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679073 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0b61d43fdb33aaaa624fc245668ccfb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"f0b61d43fdb33aaaa624fc245668ccfb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0b61d43fdb33aaaa624fc245668ccfb-config\") pod \"kube-apiserver-proxy-ip-10-0-141-9.ec2.internal\" (UID: \"f0b61d43fdb33aaaa624fc245668ccfb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.679311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.679143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c8e4874738fa63c518d68be7c33bd3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal\" (UID: \"28c8e4874738fa63c518d68be7c33bd3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.697255 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.697218    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.798405 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.798342    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.898085 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.897999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.899077 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.899052    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:18.902331 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:18.902003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:18.999334 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:18.999285    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.099950 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:19.099908    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.134279 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.134249    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:19.154991 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.154908    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:19.200489 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:19.200447    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.221669 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.221639    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 22:17:19.221846 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.221824    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:17:19.221890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.221851    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:17:19.221890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.221850    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 22:17:19.295792 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.295739    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 22:12:18 +0000 UTC" deadline="2028-04-13 14:15:32.63851819 +0000 UTC"
Jul 16 22:17:19.295792 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.295785    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15279h58m13.342737762s"
Jul 16 22:17:19.300917 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:19.300892    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.401593 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:19.401561    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.502301 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:19.502256    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-9.ec2.internal\" not found"
Jul 16 22:17:19.513059 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:19.513012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c8e4874738fa63c518d68be7c33bd3.slice/crio-6086b23d2291ec3bb422ed035a312815140c9444af0a25cc09d2c227569a0074 WatchSource:0}: Error finding container 6086b23d2291ec3bb422ed035a312815140c9444af0a25cc09d2c227569a0074: Status 404 returned error can't find the container with id 6086b23d2291ec3bb422ed035a312815140c9444af0a25cc09d2c227569a0074
Jul 16 22:17:19.518455 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.518436    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:17:19.560384 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.560349    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:19.576258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.576222    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:19.590573 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.590539    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:17:19.591503 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.591488    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal"
Jul 16 22:17:19.599549 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:19.599529    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 22:17:20.118260 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.118231    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b61d43fdb33aaaa624fc245668ccfb.slice/crio-0021ba88196fa7bdc404f6ed440678f6e11d89a7e7c1971c4ad99ba22c6019ae WatchSource:0}: Error finding container 0021ba88196fa7bdc404f6ed440678f6e11d89a7e7c1971c4ad99ba22c6019ae: Status 404 returned error can't find the container with id 0021ba88196fa7bdc404f6ed440678f6e11d89a7e7c1971c4ad99ba22c6019ae
Jul 16 22:17:20.258394 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.258355    2579 apiserver.go:52] "Watching apiserver"
Jul 16 22:17:20.264827 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.264801    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 22:17:20.265233 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.265208    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kg9h9","kube-system/konnectivity-agent-v2w7b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj","openshift-dns/node-resolver-rkx9c","openshift-image-registry/node-ca-7w9cp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal","openshift-multus/multus-additional-cni-plugins-nz844","openshift-multus/multus-dnzvd","openshift-multus/network-metrics-daemon-57ngb","kube-system/global-pull-secret-syncer-6bwd5","kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fdmt9","openshift-network-diagnostics/network-check-target-nhbx5","openshift-network-operator/iptables-alerter-44cgn"]
Jul 16 22:17:20.267047 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.267024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.268582 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.268554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:20.268680 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.268634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fj487\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 22:17:20.269812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.269760    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.270809 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.270792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.271981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.271673    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 22:17:20.271981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.271791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.271981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.271681    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lqk6p\""
Jul 16 22:17:20.271981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.271900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.272430 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.272288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.272786 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.272766    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.272890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.272857    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxtpw\""
Jul 16 22:17:20.272979 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.272953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.273434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.273414    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.274034 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.274013    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.274286 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.274251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 22:17:20.274652 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.274495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.274652 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.274562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j22mb\""
Jul 16 22:17:20.274652 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.274616    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.274822 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.274681    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:20.275530 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.275512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.276144 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.276009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.276144 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.276018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gk8jc\""
Jul 16 22:17:20.276319 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.276307    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.276780 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.276545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 22:17:20.276879 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.276789    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 22:17:20.277543 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.277525    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.279062 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.278662    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.279651 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.278634    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 22:17:20.279651 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.279350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 22:17:20.279651 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.279386    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jd654\""
Jul 16 22:17:20.279963 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.279944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.280090 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.280044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.280156 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.280026    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:20.280663 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.280643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 22:17:20.280963 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.280943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gph8l\""
Jul 16 22:17:20.281222 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.281206    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 22:17:20.281546 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.280796    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 22:17:20.282222 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.282202    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 22:17:20.282967 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.282586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.284778 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.284757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.285221 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.285148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jrp5w\""
Jul 16 22:17:20.285603 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.285436    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 22:17:20.286725 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.286683    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ds6kb\""
Jul 16 22:17:20.286835 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.286808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 22:17:20.286835 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.286817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:17:20.288042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmmj\" (UniqueName: \"kubernetes.io/projected/0f02db5c-d505-464b-8bd6-553d97dc4a47-kube-api-access-bwmmj\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.288139 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-etc-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288139 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288139 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-socket-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.288291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-sys-fs\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.288291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70592430-0a46-46b8-8650-2b3bfca5bb1e-hosts-file\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.288291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99v86\" (UniqueName: \"kubernetes.io/projected/069f7c17-92a1-44ae-9039-3008ce6bda24-kube-api-access-99v86\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.288291 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-log-socket\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-bin\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-cnibin\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-os-release\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-binary-copy\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-systemd-units\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-netns\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj65h\" (UniqueName: \"kubernetes.io/projected/e05d30bc-d843-4fd5-9024-0f09653a10cd-kube-api-access-jj65h\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-registration-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvkx\" (UniqueName: \"kubernetes.io/projected/70592430-0a46-46b8-8650-2b3bfca5bb1e-kube-api-access-4hvkx\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-config\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-env-overrides\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-script-lib\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70592430-0a46-46b8-8650-2b3bfca5bb1e-tmp-dir\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.288909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-ovn\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dph7d\" (UniqueName: \"kubernetes.io/projected/e03ee893-7de9-42e6-a8a4-90033036330a-kube-api-access-dph7d\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069f7c17-92a1-44ae-9039-3008ce6bda24-host\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7h5\" (UniqueName: \"kubernetes.io/projected/53207566-66e5-4cca-909d-754cbcb13ee1-kube-api-access-dn7h5\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.288990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-system-cni-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f02db5c-d505-464b-8bd6-553d97dc4a47-host-slash\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-slash\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/069f7c17-92a1-44ae-9039-3008ce6bda24-serviceca\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4bf\" (UniqueName: \"kubernetes.io/projected/6382ec4b-30a4-468b-acd3-e885bc88981e-kube-api-access-nj4bf\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f02db5c-d505-464b-8bd6-553d97dc4a47-iptables-alerter-script\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-kubelet\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-systemd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-var-lib-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.289433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-node-log\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.290041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-netd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.290041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.289344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-device-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.291509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.291485    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 22:17:20.310882 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.310858    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s62dh"
Jul 16 22:17:20.318391 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.318369    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s62dh"
Jul 16 22:17:20.377005 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.376921    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 22:17:20.389734 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f02db5c-d505-464b-8bd6-553d97dc4a47-host-slash\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-slash\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/069f7c17-92a1-44ae-9039-3008ce6bda24-serviceca\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-multus\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-slash\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.389914 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f02db5c-d505-464b-8bd6-553d97dc4a47-host-slash\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.389989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f02db5c-d505-464b-8bd6-553d97dc4a47-iptables-alerter-script\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-kubelet\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-node-log\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-device-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-kubelet\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-daemon-config\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-node-log\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-konnectivity-ca\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-kubelet-config\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-etc-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-socket-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-log-socket\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390243    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-etc-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-cnibin\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-device-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-os-release\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-log-socket\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-binary-copy\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-socket-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/069f7c17-92a1-44ae-9039-3008ce6bda24-serviceca\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-cnibin\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-systemd-units\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-os-release\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390416    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvkx\" (UniqueName: \"kubernetes.io/projected/70592430-0a46-46b8-8650-2b3bfca5bb1e-kube-api-access-4hvkx\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.390752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-systemd-units\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cnibin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0f02db5c-d505-464b-8bd6-553d97dc4a47-iptables-alerter-script\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-hostroot\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-run\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70592430-0a46-46b8-8650-2b3bfca5bb1e-tmp-dir\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-var-lib-kubelet\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-ovn\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.390983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dph7d\" (UniqueName: \"kubernetes.io/projected/e03ee893-7de9-42e6-a8a4-90033036330a-kube-api-access-dph7d\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-etc-kubernetes\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.391496 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.391022    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm5kb\" (UniqueName: \"kubernetes.io/projected/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-kube-api-access-gm5kb\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-modprobe-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.391093    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:20.891072771 +0000 UTC m=+2.959395804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70592430-0a46-46b8-8650-2b3bfca5bb1e-tmp-dir\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cni-binary-copy\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-multus-certs\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysconfig\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-kubernetes\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-binary-copy\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf4jz\" (UniqueName: \"kubernetes.io/projected/da6f4633-d0db-4ded-a553-eff192dfa6f1-kube-api-access-tf4jz\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-ovn\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4bf\" (UniqueName: \"kubernetes.io/projected/6382ec4b-30a4-468b-acd3-e885bc88981e-kube-api-access-nj4bf\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-systemd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-var-lib-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.392370 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-run-systemd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-netd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99v86\" (UniqueName: \"kubernetes.io/projected/069f7c17-92a1-44ae-9039-3008ce6bda24-kube-api-access-99v86\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-var-lib-openvswitch\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-netd\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-lib-modules\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmmj\" (UniqueName: \"kubernetes.io/projected/0f02db5c-d505-464b-8bd6-553d97dc4a47-kube-api-access-bwmmj\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-sys-fs\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70592430-0a46-46b8-8650-2b3bfca5bb1e-hosts-file\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-netns\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-host\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-sys-fs\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/70592430-0a46-46b8-8650-2b3bfca5bb1e-hosts-file\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-tmp\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393135 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.391980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj65h\" (UniqueName: \"kubernetes.io/projected/e05d30bc-d843-4fd5-9024-0f09653a10cd-kube-api-access-jj65h\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-registration-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-dbus\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-bin\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-sys\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-tuned\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-netns\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392159    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392200    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-cni-bin\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e03ee893-7de9-42e6-a8a4-90033036330a-registration-dir\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-netns\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-os-release\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-conf-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-config\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-env-overrides\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.393911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-script-lib\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-system-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-socket-dir-parent\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-bin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-conf\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-k8s-cni-cncf-io\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-kubelet\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-systemd\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069f7c17-92a1-44ae-9039-3008ce6bda24-host\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.392979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7h5\" (UniqueName: \"kubernetes.io/projected/53207566-66e5-4cca-909d-754cbcb13ee1-kube-api-access-dn7h5\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-agent-certs\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-system-cni-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-config\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-env-overrides\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.394495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovnkube-script-lib\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.395109 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e05d30bc-d843-4fd5-9024-0f09653a10cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.395109 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/069f7c17-92a1-44ae-9039-3008ce6bda24-host\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.395109 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6382ec4b-30a4-468b-acd3-e885bc88981e-system-cni-dir\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.395109 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.393730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6382ec4b-30a4-468b-acd3-e885bc88981e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.396599 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.396572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e05d30bc-d843-4fd5-9024-0f09653a10cd-ovn-node-metrics-cert\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.400211 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.399685    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:20.400211 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.399782    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:20.400211 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.399799    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:20.400211 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.399896    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:20.8998724 +0000 UTC m=+2.968195450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:20.400680 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.400661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99v86\" (UniqueName: \"kubernetes.io/projected/069f7c17-92a1-44ae-9039-3008ce6bda24-kube-api-access-99v86\") pod \"node-ca-7w9cp\" (UID: \"069f7c17-92a1-44ae-9039-3008ce6bda24\") " pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.403591 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.403560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dph7d\" (UniqueName: \"kubernetes.io/projected/e03ee893-7de9-42e6-a8a4-90033036330a-kube-api-access-dph7d\") pod \"aws-ebs-csi-driver-node-ll2rj\" (UID: \"e03ee893-7de9-42e6-a8a4-90033036330a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.404293 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.404266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvkx\" (UniqueName: \"kubernetes.io/projected/70592430-0a46-46b8-8650-2b3bfca5bb1e-kube-api-access-4hvkx\") pod \"node-resolver-rkx9c\" (UID: \"70592430-0a46-46b8-8650-2b3bfca5bb1e\") " pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.404293 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.404285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmmj\" (UniqueName: \"kubernetes.io/projected/0f02db5c-d505-464b-8bd6-553d97dc4a47-kube-api-access-bwmmj\") pod \"iptables-alerter-44cgn\" (UID: \"0f02db5c-d505-464b-8bd6-553d97dc4a47\") " pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.404446 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.404291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4bf\" (UniqueName: \"kubernetes.io/projected/6382ec4b-30a4-468b-acd3-e885bc88981e-kube-api-access-nj4bf\") pod \"multus-additional-cni-plugins-nz844\" (UID: \"6382ec4b-30a4-468b-acd3-e885bc88981e\") " pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.405851 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.405821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj65h\" (UniqueName: \"kubernetes.io/projected/e05d30bc-d843-4fd5-9024-0f09653a10cd-kube-api-access-jj65h\") pod \"ovnkube-node-kg9h9\" (UID: \"e05d30bc-d843-4fd5-9024-0f09653a10cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.406376 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.406349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7h5\" (UniqueName: \"kubernetes.io/projected/53207566-66e5-4cca-909d-754cbcb13ee1-kube-api-access-dn7h5\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.459813 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.459756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" event={"ID":"f0b61d43fdb33aaaa624fc245668ccfb","Type":"ContainerStarted","Data":"0021ba88196fa7bdc404f6ed440678f6e11d89a7e7c1971c4ad99ba22c6019ae"}
Jul 16 22:17:20.461081 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.461053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"28c8e4874738fa63c518d68be7c33bd3","Type":"ContainerStarted","Data":"6086b23d2291ec3bb422ed035a312815140c9444af0a25cc09d2c227569a0074"}
Jul 16 22:17:20.493580 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-lib-modules\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-netns\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-host\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-tmp\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-dbus\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-sys\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-netns\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-tuned\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.493830 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-os-release\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-conf-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-dbus\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-system-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-system-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-socket-dir-parent\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-host\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-bin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-conf\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-cni-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-k8s-cni-cncf-io\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-kubelet\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-os-release\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-systemd\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-conf-dir\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-agent-certs\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-multus\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.494299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-sys\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.493722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-lib-modules\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-daemon-config\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-konnectivity-ca\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-kubelet-config\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-kubelet\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysctl-conf\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-k8s-cni-cncf-io\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-multus\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cnibin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-hostroot\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-run\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-var-lib-kubelet\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cnibin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-etc-kubernetes\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-hostroot\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-var-lib-cni-bin\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.495149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-var-lib-kubelet\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-etc-kubernetes\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm5kb\" (UniqueName: \"kubernetes.io/projected/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-kube-api-access-gm5kb\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.494507    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-modprobe-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-socket-dir-parent\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cni-binary-copy\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.494565    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:20.994546926 +0000 UTC m=+3.062869960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-multus-certs\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysconfig\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-kubernetes\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf4jz\" (UniqueName: \"kubernetes.io/projected/da6f4633-d0db-4ded-a553-eff192dfa6f1-kube-api-access-tf4jz\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-multus-daemon-config\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-run\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-konnectivity-ca\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-modprobe-d\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-systemd\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/78b06a78-a819-426c-8aee-a671d9a09e6b-kubelet-config\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:20.496749 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-host-run-multus-certs\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.496749 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-sysconfig\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496749 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.494929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-kubernetes\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.496749 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.495587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-cni-binary-copy\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.497163 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.497143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-tmp\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.497234 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.497176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/da6f4633-d0db-4ded-a553-eff192dfa6f1-etc-tuned\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.497865 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.497842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a-agent-certs\") pod \"konnectivity-agent-v2w7b\" (UID: \"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a\") " pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.504505 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.504480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf4jz\" (UniqueName: \"kubernetes.io/projected/da6f4633-d0db-4ded-a553-eff192dfa6f1-kube-api-access-tf4jz\") pod \"tuned-fdmt9\" (UID: \"da6f4633-d0db-4ded-a553-eff192dfa6f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.504637 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.504609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm5kb\" (UniqueName: \"kubernetes.io/projected/de6c1ef5-1296-409d-8d4e-e0a180b14f1b-kube-api-access-gm5kb\") pod \"multus-dnzvd\" (UID: \"de6c1ef5-1296-409d-8d4e-e0a180b14f1b\") " pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.581700 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.581663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nz844"
Jul 16 22:17:20.588152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.588116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj"
Jul 16 22:17:20.597269 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.597231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkx9c"
Jul 16 22:17:20.605159 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.605136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7w9cp"
Jul 16 22:17:20.612051 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.612012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44cgn"
Jul 16 22:17:20.617981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.617953    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:20.618338 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.618317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:20.625878 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.625857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:20.632576 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.632549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dnzvd"
Jul 16 22:17:20.638414 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.638383    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fdmt9"
Jul 16 22:17:20.866868 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.866837    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6c1ef5_1296_409d_8d4e_e0a180b14f1b.slice/crio-5877f2e28ab55029c1c2a4e25fe48636763e401503d069c5e9de68cef8c105aa WatchSource:0}: Error finding container 5877f2e28ab55029c1c2a4e25fe48636763e401503d069c5e9de68cef8c105aa: Status 404 returned error can't find the container with id 5877f2e28ab55029c1c2a4e25fe48636763e401503d069c5e9de68cef8c105aa
Jul 16 22:17:20.868530 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.868498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7d03e2_0bf9_4bcf_84bb_fa7b7c8a0a2a.slice/crio-54dcc08f77a5d4e750c94f46b071a307e301753681959a11708c491ce27ba904 WatchSource:0}: Error finding container 54dcc08f77a5d4e750c94f46b071a307e301753681959a11708c491ce27ba904: Status 404 returned error can't find the container with id 54dcc08f77a5d4e750c94f46b071a307e301753681959a11708c491ce27ba904
Jul 16 22:17:20.873450 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.873423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05d30bc_d843_4fd5_9024_0f09653a10cd.slice/crio-d6cb5fd095176cf80e3d1d516be2af6f640e59e5a749c7ab3becbd6bbb3fe89b WatchSource:0}: Error finding container d6cb5fd095176cf80e3d1d516be2af6f640e59e5a749c7ab3becbd6bbb3fe89b: Status 404 returned error can't find the container with id d6cb5fd095176cf80e3d1d516be2af6f640e59e5a749c7ab3becbd6bbb3fe89b
Jul 16 22:17:20.876861 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.876839    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069f7c17_92a1_44ae_9039_3008ce6bda24.slice/crio-888f15be2e4a7e70449903bac344e8b1128bc0231faf1e87719a25cc7f927c0f WatchSource:0}: Error finding container 888f15be2e4a7e70449903bac344e8b1128bc0231faf1e87719a25cc7f927c0f: Status 404 returned error can't find the container with id 888f15be2e4a7e70449903bac344e8b1128bc0231faf1e87719a25cc7f927c0f
Jul 16 22:17:20.877916 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.877840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f02db5c_d505_464b_8bd6_553d97dc4a47.slice/crio-fd12dd5ca38e5fc076a451eadf09508c39a38a6bc3d9f5169387cb59082516b9 WatchSource:0}: Error finding container fd12dd5ca38e5fc076a451eadf09508c39a38a6bc3d9f5169387cb59082516b9: Status 404 returned error can't find the container with id fd12dd5ca38e5fc076a451eadf09508c39a38a6bc3d9f5169387cb59082516b9
Jul 16 22:17:20.878564 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.878499    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6f4633_d0db_4ded_a553_eff192dfa6f1.slice/crio-3c59c7eeeb60df3f6513ccd25a7ade43888fdb7e96d19374708fba2875a2dda3 WatchSource:0}: Error finding container 3c59c7eeeb60df3f6513ccd25a7ade43888fdb7e96d19374708fba2875a2dda3: Status 404 returned error can't find the container with id 3c59c7eeeb60df3f6513ccd25a7ade43888fdb7e96d19374708fba2875a2dda3
Jul 16 22:17:20.879524 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.879373    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03ee893_7de9_42e6_a8a4_90033036330a.slice/crio-d55b71d6328021b7ee5d4ae84389f51a4f6a87a8dbd7e80eabf94a50fd0b1596 WatchSource:0}: Error finding container d55b71d6328021b7ee5d4ae84389f51a4f6a87a8dbd7e80eabf94a50fd0b1596: Status 404 returned error can't find the container with id d55b71d6328021b7ee5d4ae84389f51a4f6a87a8dbd7e80eabf94a50fd0b1596
Jul 16 22:17:20.880398 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.880223    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70592430_0a46_46b8_8650_2b3bfca5bb1e.slice/crio-15a395fee04920f220c5d8d02b3f54b793b2a76d628fb4086255ac2f1744d734 WatchSource:0}: Error finding container 15a395fee04920f220c5d8d02b3f54b793b2a76d628fb4086255ac2f1744d734: Status 404 returned error can't find the container with id 15a395fee04920f220c5d8d02b3f54b793b2a76d628fb4086255ac2f1744d734
Jul 16 22:17:20.881894 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:17:20.881872    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6382ec4b_30a4_468b_acd3_e885bc88981e.slice/crio-f9345b681d8f7fbeeeb9c411880d5a639b7ed9bc7950d554da68ef3c17ebf29a WatchSource:0}: Error finding container f9345b681d8f7fbeeeb9c411880d5a639b7ed9bc7950d554da68ef3c17ebf29a: Status 404 returned error can't find the container with id f9345b681d8f7fbeeeb9c411880d5a639b7ed9bc7950d554da68ef3c17ebf29a
Jul 16 22:17:20.898499 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.898347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:20.898557 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.898504    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:20.898590 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:20.898568    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:21.898549547 +0000 UTC m=+3.966872581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:20.999965 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:20.999907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:21.000226 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.000061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:21.000226 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000149    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:21.000226 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000171    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:21.000226 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000183    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:21.000226 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000221    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:21.000541 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:22.00024162 +0000 UTC m=+4.068564677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:21.000541 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.000336    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:22.000323527 +0000 UTC m=+4.068646576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:21.095238 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.095134    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:21.321280 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.321179    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:12:20 +0000 UTC" deadline="2028-01-01 17:01:41.567758115 +0000 UTC"
Jul 16 22:17:21.321280 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.321250    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12810h44m20.246513187s"
Jul 16 22:17:21.457239 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.456593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:21.457239 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.456759    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:21.467504 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.467393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj" event={"ID":"e03ee893-7de9-42e6-a8a4-90033036330a","Type":"ContainerStarted","Data":"d55b71d6328021b7ee5d4ae84389f51a4f6a87a8dbd7e80eabf94a50fd0b1596"}
Jul 16 22:17:21.472288 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.472176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44cgn" event={"ID":"0f02db5c-d505-464b-8bd6-553d97dc4a47","Type":"ContainerStarted","Data":"fd12dd5ca38e5fc076a451eadf09508c39a38a6bc3d9f5169387cb59082516b9"}
Jul 16 22:17:21.474858 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.474820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"d6cb5fd095176cf80e3d1d516be2af6f640e59e5a749c7ab3becbd6bbb3fe89b"}
Jul 16 22:17:21.476505 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.476477    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnzvd" event={"ID":"de6c1ef5-1296-409d-8d4e-e0a180b14f1b","Type":"ContainerStarted","Data":"5877f2e28ab55029c1c2a4e25fe48636763e401503d069c5e9de68cef8c105aa"}
Jul 16 22:17:21.478618 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.478581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fdmt9" event={"ID":"da6f4633-d0db-4ded-a553-eff192dfa6f1","Type":"ContainerStarted","Data":"3c59c7eeeb60df3f6513ccd25a7ade43888fdb7e96d19374708fba2875a2dda3"}
Jul 16 22:17:21.482394 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.482362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7w9cp" event={"ID":"069f7c17-92a1-44ae-9039-3008ce6bda24","Type":"ContainerStarted","Data":"888f15be2e4a7e70449903bac344e8b1128bc0231faf1e87719a25cc7f927c0f"}
Jul 16 22:17:21.490877 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.490841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2w7b" event={"ID":"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a","Type":"ContainerStarted","Data":"54dcc08f77a5d4e750c94f46b071a307e301753681959a11708c491ce27ba904"}
Jul 16 22:17:21.493856 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.493822    2579 generic.go:356] "Generic (PLEG): container finished" podID="28c8e4874738fa63c518d68be7c33bd3" containerID="51574778a4a1a4b8c0981c97b820a9d3d038e9a42685e601c5177b5e20250bf4" exitCode=0
Jul 16 22:17:21.494187 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.494123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"28c8e4874738fa63c518d68be7c33bd3","Type":"ContainerDied","Data":"51574778a4a1a4b8c0981c97b820a9d3d038e9a42685e601c5177b5e20250bf4"}
Jul 16 22:17:21.497992 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.497933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerStarted","Data":"f9345b681d8f7fbeeeb9c411880d5a639b7ed9bc7950d554da68ef3c17ebf29a"}
Jul 16 22:17:21.499616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.499549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkx9c" event={"ID":"70592430-0a46-46b8-8650-2b3bfca5bb1e","Type":"ContainerStarted","Data":"15a395fee04920f220c5d8d02b3f54b793b2a76d628fb4086255ac2f1744d734"}
Jul 16 22:17:21.911466 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:21.911336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:21.911690 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.911570    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:21.911690 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:21.911640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:23.911618388 +0000 UTC m=+5.979941432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.012763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.012835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.012997    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.013062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:24.013040382 +0000 UTC m=+6.081363419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.013498    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.013520    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.013533    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:22.013608 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.013584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:24.013568595 +0000 UTC m=+6.081891664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:22.130665 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.130313    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 22:17:22.321804 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.321681    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 22:12:20 +0000 UTC" deadline="2027-12-18 00:59:25.585572372 +0000 UTC"
Jul 16 22:17:22.321804 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.321739    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12458h42m3.263838799s"
Jul 16 22:17:22.461246 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.460363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:22.461246 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.460544    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:22.461246 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.460993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:22.461246 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:22.461099    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:22.525231 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:22.525191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" event={"ID":"28c8e4874738fa63c518d68be7c33bd3","Type":"ContainerStarted","Data":"ab376744090b692f55eb08e63746eb4609fad82a49e126cd9e1cef574fbba8d3"}
Jul 16 22:17:23.456731 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:23.456628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:23.457609 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:23.456959    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:23.931059 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:23.930345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:23.931059 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:23.930527    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:23.931059 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:23.930598    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:27.930579714 +0000 UTC m=+9.998902751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:24.031621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:24.031772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.031954    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.031977    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.031992    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.032062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:28.032044775 +0000 UTC m=+10.100367821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.032473    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:24.032571 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.032532    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:28.032516094 +0000 UTC m=+10.100839131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:24.456525 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:24.456405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:24.456748 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.456543    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:24.456748 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:24.456592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:24.457195 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:24.456747    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:25.456160 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:25.456127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:25.456347 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:25.456281    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:26.456864 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:26.456825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:26.457348 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:26.456976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:26.457348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:26.456825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:26.457473 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:26.457410    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:27.456185 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:27.456146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:27.456404 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:27.456312    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:27.966535 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:27.966494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:27.967056 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:27.966645    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:27.967056 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:27.966742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:35.966719594 +0000 UTC m=+18.035042641 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:28.068160 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:28.067674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:28.068160 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:28.067737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:28.068160 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.067835    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:28.068160 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.067882    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:36.067870111 +0000 UTC m=+18.136193143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:28.068160 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.068157    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:28.068623 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.068181    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:28.068623 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.068193    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:28.068623 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.068252    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:36.068228354 +0000 UTC m=+18.136551389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:28.458033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:28.457398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:28.458033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:28.457445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:28.458033 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.457541    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:28.458033 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:28.457734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:29.456932 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:29.456719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:29.456932 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:29.456870    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:30.456196 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:30.456111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:30.456399 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:30.456265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:30.456821 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:30.456618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:30.456821 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:30.456755    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:31.455894 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:31.455861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:31.456383 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:31.456020    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:32.456874 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:32.456833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:32.457374 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:32.456888    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:32.457374 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:32.456988    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:32.457374 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:32.457128    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:33.456091 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:33.456058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:33.456288 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:33.456229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:34.456336 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:34.456299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:34.456795 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:34.456299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:34.456795 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:34.456425    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:34.456795 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:34.456521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:35.456861 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:35.456825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:35.457241 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:35.456949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:36.029915 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:36.029880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:36.030088 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.030038    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:36.030137 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.030094    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:52.030079151 +0000 UTC m=+34.098402184 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:36.130282 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:36.130243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:36.130473 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:36.130313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:36.130473 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130418    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:36.130577 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130480    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:17:52.130461356 +0000 UTC m=+34.198784409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:36.130577 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130417    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:36.130577 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130512    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:36.130577 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130528    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:36.130808 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.130581    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:17:52.130563394 +0000 UTC m=+34.198886480 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:36.456956 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:36.456855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:36.456956 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:36.456875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:36.457535 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.457009    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:36.457535 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:36.457078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:37.456546 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:37.456517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:37.456738 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:37.456623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:38.456910 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:38.456867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:38.457361 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:38.456971    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:38.457361 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:38.457055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:38.457361 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:38.457166    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:39.456451 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:39.456408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:39.456647 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:39.456549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:40.456762 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.456479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:40.457231 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.456507    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:40.457231 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:40.456879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:40.457231 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:40.456942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:40.560886 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.560856    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="8824c8b603080e0dcf9186ee9fbbaba98fd1f762cec6874e9fc302b38634c85d" exitCode=0
Jul 16 22:17:40.561065 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.560946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"8824c8b603080e0dcf9186ee9fbbaba98fd1f762cec6874e9fc302b38634c85d"}
Jul 16 22:17:40.562421 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.562363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkx9c" event={"ID":"70592430-0a46-46b8-8650-2b3bfca5bb1e","Type":"ContainerStarted","Data":"1f289188f285ede0a6eec51d06b1ecf9b38b5f651c26cecbc26bd0c56339587c"}
Jul 16 22:17:40.563837 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.563813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj" event={"ID":"e03ee893-7de9-42e6-a8a4-90033036330a","Type":"ContainerStarted","Data":"0597e1e5adafa58b5ff02e56fddf5c22f1c3a2e6a1855ce3038a960965dc6930"}
Jul 16 22:17:40.565243 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.565218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dnzvd" event={"ID":"de6c1ef5-1296-409d-8d4e-e0a180b14f1b","Type":"ContainerStarted","Data":"cf2924960a9ff5be54291a46fb67b813f6589a11d422cbd5394febc90f31a844"}
Jul 16 22:17:40.566603 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.566583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" event={"ID":"f0b61d43fdb33aaaa624fc245668ccfb","Type":"ContainerStarted","Data":"492e38e3fe11c95565a65c8545008a0d90354a6ad12117c57c0c278fe0c81d0e"}
Jul 16 22:17:40.567957 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.567936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fdmt9" event={"ID":"da6f4633-d0db-4ded-a553-eff192dfa6f1","Type":"ContainerStarted","Data":"f1ced64573fec39a76c5508bd303705aa85fd810a742a0db0a5d11f01a547c0e"}
Jul 16 22:17:40.570724 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.570686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7w9cp" event={"ID":"069f7c17-92a1-44ae-9039-3008ce6bda24","Type":"ContainerStarted","Data":"15bd1fba0a48ee1b2f765bac86e1c0fa871277c70a1f91ee1a6117ee7b7359fc"}
Jul 16 22:17:40.572491 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.572473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v2w7b" event={"ID":"fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a","Type":"ContainerStarted","Data":"1bcdc0836bcbb25043d24f4a08d3db08be6869c6956cfe838cc321fe5d1c71a0"}
Jul 16 22:17:40.591365 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.591318    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-9.ec2.internal" podStartSLOduration=21.591305949 podStartE2EDuration="21.591305949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:22.542925074 +0000 UTC m=+4.611248131" watchObservedRunningTime="2026-07-16 22:17:40.591305949 +0000 UTC m=+22.659629003"
Jul 16 22:17:40.608140 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.608091    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-9.ec2.internal" podStartSLOduration=21.608075849 podStartE2EDuration="21.608075849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.608058131 +0000 UTC m=+22.676381195" watchObservedRunningTime="2026-07-16 22:17:40.608075849 +0000 UTC m=+22.676398903"
Jul 16 22:17:40.627215 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.627159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7w9cp" podStartSLOduration=3.40402134 podStartE2EDuration="22.627136012s" totalImagesPullingTime="19.223114672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.626387254 +0000 UTC m=+22.694710308" watchObservedRunningTime="2026-07-16 22:17:40.627136012 +0000 UTC m=+22.695459069"
Jul 16 22:17:40.665665 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.665627    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v2w7b" podStartSLOduration=3.435560563 podStartE2EDuration="22.665611813s" totalImagesPullingTime="19.23005125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.643121401 +0000 UTC m=+22.711444454" watchObservedRunningTime="2026-07-16 22:17:40.665611813 +0000 UTC m=+22.733934858"
Jul 16 22:17:40.684821 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.684774    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dnzvd" podStartSLOduration=3.437761106 podStartE2EDuration="22.68476104s" totalImagesPullingTime="19.246999934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.665927099 +0000 UTC m=+22.734250153" watchObservedRunningTime="2026-07-16 22:17:40.68476104 +0000 UTC m=+22.753084093"
Jul 16 22:17:40.709310 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.709261    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rkx9c" podStartSLOduration=3.506353966 podStartE2EDuration="22.709247348s" totalImagesPullingTime="19.202893382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.684629952 +0000 UTC m=+22.752953005" watchObservedRunningTime="2026-07-16 22:17:40.709247348 +0000 UTC m=+22.777570472"
Jul 16 22:17:40.709570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:40.709551    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fdmt9" podStartSLOduration=3.486224687 podStartE2EDuration="22.709546798s" totalImagesPullingTime="19.223322111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.709332242 +0000 UTC m=+22.777655322" watchObservedRunningTime="2026-07-16 22:17:40.709546798 +0000 UTC m=+22.777869851"
Jul 16 22:17:41.456231 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.455976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:41.456474 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:41.456345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:41.575993 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.575953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44cgn" event={"ID":"0f02db5c-d505-464b-8bd6-553d97dc4a47","Type":"ContainerStarted","Data":"84146de5d30d73dbb7607b63d524eb3934515480debb5f533c1badc231f081f1"}
Jul 16 22:17:41.577603 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.577577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"cf441b97d4fd4f420c143f56812812687d21a447649abb65f340ab50b1b0dd1a"}
Jul 16 22:17:41.577732 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.577650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"9347a3111bad8985ed4f2cb2c9a9fea999d1b8677ea58c523db434c248aae2a1"}
Jul 16 22:17:41.590385 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.590285    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44cgn" podStartSLOduration=4.367406084 podStartE2EDuration="23.590198283s" totalImagesPullingTime="19.222792199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:41.589540335 +0000 UTC m=+23.657863388" watchObservedRunningTime="2026-07-16 22:17:41.590198283 +0000 UTC m=+23.658521338"
Jul 16 22:17:41.877896 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:41.877863    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 22:17:42.353792 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.353681    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T22:17:41.877922442Z","UUID":"bfe340c7-bec3-428f-a7c8-ddb6162cb1cf","Handler":null,"Name":"","Endpoint":""}
Jul 16 22:17:42.355524 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.355493    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 22:17:42.355650 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.355533    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 22:17:42.456323 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.456296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:42.456509 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.456296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:42.456509 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:42.456389    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:42.456509 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:42.456467    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:42.583634 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.583593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj" event={"ID":"e03ee893-7de9-42e6-a8a4-90033036330a","Type":"ContainerStarted","Data":"956762e930237f99eb898d3f7f00b9566d88302922e2e663a1130351ae26bf8c"}
Jul 16 22:17:42.586909 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.586881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"0cfe8c68df738f91569abaeb703e2b6e132530662297a22a75233db216fe09ce"}
Jul 16 22:17:42.587083 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.586965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"47b90b7972a98f495daee38a68a29f753410574b68ed615b6b4bd26d9fd656ab"}
Jul 16 22:17:42.587083 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.587017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"4c0fe2cc61bae438632d6e8ba6625ea70f6615fe679139039632610fe031e8be"}
Jul 16 22:17:42.587083 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:42.587070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"8f87ec39ba7c2b938ef990da19119057922b14070e567d5442917e71fb146906"}
Jul 16 22:17:43.456760 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:43.456725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:43.457034 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:43.456976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:44.456680 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.456436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:44.457167 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.456493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:44.457167 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:44.456755    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:44.457167 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:44.456821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:44.567887 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.567853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:44.568476 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.568455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:44.594415 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.594384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj" event={"ID":"e03ee893-7de9-42e6-a8a4-90033036330a","Type":"ContainerStarted","Data":"aea2ba8212e55399f6a01f7aa97a6fb913ddf263f6db9b608b4d80836c83da1c"}
Jul 16 22:17:44.598129 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.598101    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"71b9cf2b79ef244b20c8b5ad81c9d8150337df2cc88f7c4c5c5cd925d9cf8fe1"}
Jul 16 22:17:44.598256 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.598189    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:44.598686 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.598667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v2w7b"
Jul 16 22:17:44.611067 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:44.611026    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll2rj" podStartSLOduration=3.848089588 podStartE2EDuration="26.611013149s" totalImagesPullingTime="22.762923561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:44.610647946 +0000 UTC m=+26.678970999" watchObservedRunningTime="2026-07-16 22:17:44.611013149 +0000 UTC m=+26.679336202"
Jul 16 22:17:45.456452 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:45.456417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:45.456632 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:45.456543    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:46.457002 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.456580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:46.457002 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.456589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:46.457002 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:46.456922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:46.457002 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:46.456982    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:46.603833 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.603799    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="5cf6ed8470e017ca0afcf0a0c743199b7a5ac737c1afbae1d2df190057c8e388" exitCode=0
Jul 16 22:17:46.604022 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.603884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"5cf6ed8470e017ca0afcf0a0c743199b7a5ac737c1afbae1d2df190057c8e388"}
Jul 16 22:17:46.607725 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.607656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" event={"ID":"e05d30bc-d843-4fd5-9024-0f09653a10cd","Type":"ContainerStarted","Data":"8b34e7e7847773109018056da620c2d7118c3532aef85f27e725619f921af62a"}
Jul 16 22:17:46.607906 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.607888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:46.608026 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.607957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:46.625139 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.625118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:46.652210 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:46.652171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9" podStartSLOduration=8.481737404 podStartE2EDuration="28.652158776s" totalImagesPullingTime="20.170421372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:46.652065301 +0000 UTC m=+28.720388355" watchObservedRunningTime="2026-07-16 22:17:46.652158776 +0000 UTC m=+28.720481829"
Jul 16 22:17:47.456006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:47.455979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:47.456164 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:47.456082    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:47.610531 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:47.610502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:47.625159 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:47.625137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:17:48.456858 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:48.456824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:48.457067 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:48.456900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:48.457067 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:48.456988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:48.457184 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:48.457066    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:48.614013 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:48.613982    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="851928bdcbd72249514555b751b621051480cc11d49567adb10f041032b8bb00" exitCode=0
Jul 16 22:17:48.614433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:48.614061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"851928bdcbd72249514555b751b621051480cc11d49567adb10f041032b8bb00"}
Jul 16 22:17:49.456786 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:49.456755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:49.456935 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:49.456856    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:50.456422 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:50.456398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:50.456793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:50.456402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:50.456793 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:50.456491    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:50.456793 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:50.456566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:50.619790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:50.619765    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="10668de92a6f5583951850d3208955136e72212442b69444f7d835e6c8a71ab7" exitCode=0
Jul 16 22:17:50.619915 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:50.619813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"10668de92a6f5583951850d3208955136e72212442b69444f7d835e6c8a71ab7"}
Jul 16 22:17:51.456225 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:51.456184    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:51.456408 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:51.456303    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:52.043044 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:52.043005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:52.043501 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.043179    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:52.043501 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.043256    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs podName:53207566-66e5-4cca-909d-754cbcb13ee1 nodeName:}" failed. No retries permitted until 2026-07-16 22:18:24.043235022 +0000 UTC m=+66.111558081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs") pod "network-metrics-daemon-57ngb" (UID: "53207566-66e5-4cca-909d-754cbcb13ee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 22:17:52.143515 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:52.143486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:52.143671 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:52.143547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:52.143671 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143640    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:52.143671 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143656    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 22:17:52.143671 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143671    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 22:17:52.143882 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143685    2579 projected.go:196] Error preparing data for projected volume kube-api-access-dt6hh for pod openshift-network-diagnostics/network-check-target-nhbx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:52.143882 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143726    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret podName:78b06a78-a819-426c-8aee-a671d9a09e6b nodeName:}" failed. No retries permitted until 2026-07-16 22:18:24.143692371 +0000 UTC m=+66.212015404 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret") pod "global-pull-secret-syncer-6bwd5" (UID: "78b06a78-a819-426c-8aee-a671d9a09e6b") : object "kube-system"/"original-pull-secret" not registered
Jul 16 22:17:52.143882 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.143746    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh podName:ede76fdf-2fb3-4493-8965-39daec8e3a86 nodeName:}" failed. No retries permitted until 2026-07-16 22:18:24.143735025 +0000 UTC m=+66.212058056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dt6hh" (UniqueName: "kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh") pod "network-check-target-nhbx5" (UID: "ede76fdf-2fb3-4493-8965-39daec8e3a86") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 22:17:52.456490 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:52.456464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:52.456644 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.456582    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:52.456644 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:52.456633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:52.456790 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:52.456746    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:53.456076 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:53.456041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:53.456465 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:53.456154    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:54.456201 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:54.456170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:54.456665 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:54.456304    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:54.456665 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:54.456346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:54.456665 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:54.456462    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:55.456843 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:55.456654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:55.457192 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:55.456918    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:55.634889 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:55.634860    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="9d9f2da4cd478c1d7a4e07dde63cd28f706acce90391bd9175584af83f0c44a5" exitCode=0
Jul 16 22:17:55.635021 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:55.634915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"9d9f2da4cd478c1d7a4e07dde63cd28f706acce90391bd9175584af83f0c44a5"}
Jul 16 22:17:56.456326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:56.456292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:56.456515 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:56.456382    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:56.456515 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:56.456436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:56.456515 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:56.456511    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:56.639806 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:56.639778    2579 generic.go:356] "Generic (PLEG): container finished" podID="6382ec4b-30a4-468b-acd3-e885bc88981e" containerID="1bc0b44c3b8620d80486ca24e8becd8e7041f7d9f0050553b4b20db1e1c4537c" exitCode=0
Jul 16 22:17:56.640146 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:56.639835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerDied","Data":"1bc0b44c3b8620d80486ca24e8becd8e7041f7d9f0050553b4b20db1e1c4537c"}
Jul 16 22:17:57.455871 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:57.455845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:57.456019 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:57.455950    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:17:57.645041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:57.645012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nz844" event={"ID":"6382ec4b-30a4-468b-acd3-e885bc88981e","Type":"ContainerStarted","Data":"dcfe7d756d8157d3613ccd489f9cfc947903d2e8cd8821f38df75dfd1bf5f0e8"}
Jul 16 22:17:57.670220 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:57.670179    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nz844" podStartSLOduration=8.390281986 podStartE2EDuration="39.670167845s" totalImagesPullingTime="31.279885859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:57.66999078 +0000 UTC m=+39.738313834" watchObservedRunningTime="2026-07-16 22:17:57.670167845 +0000 UTC m=+39.738490893"
Jul 16 22:17:58.457449 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:58.457422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:17:58.457687 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:58.457494    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:17:58.457687 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:58.457562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:17:58.457687 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:58.457657    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:17:59.456670 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:17:59.456638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:17:59.457056 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:17:59.456749    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:18:00.456485 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:00.456456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:00.456644 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:00.456550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:18:00.456644 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:00.456624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:00.456744 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:00.456721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:18:01.456246 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.456215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:01.456339 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:01.456321    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:18:01.632575 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.632540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57ngb"]
Jul 16 22:18:01.633231 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.633138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhbx5"]
Jul 16 22:18:01.633296 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.633269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:01.633492 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:01.633450    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:18:01.633888 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.633869    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bwd5"]
Jul 16 22:18:01.633988 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.633974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:01.634090 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:01.634073    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:18:01.653720 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:01.653646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:01.653823 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:01.653799    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:18:03.456752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:03.456723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:03.457104 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:03.456731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:03.457104 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:03.456828    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-57ngb" podUID="53207566-66e5-4cca-909d-754cbcb13ee1"
Jul 16 22:18:03.457104 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:03.456836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:03.457104 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:03.456900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6bwd5" podUID="78b06a78-a819-426c-8aee-a671d9a09e6b"
Jul 16 22:18:03.457104 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:18:03.456941    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nhbx5" podUID="ede76fdf-2fb3-4493-8965-39daec8e3a86"
Jul 16 22:18:05.252461 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.252430    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-9.ec2.internal" event="NodeReady"
Jul 16 22:18:05.252886 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.252548    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 22:18:05.304641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.304611    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t7wwr"]
Jul 16 22:18:05.330121 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.330088    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t5smt"]
Jul 16 22:18:05.330258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.330201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.332701 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.332492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ssz9k\""
Jul 16 22:18:05.332840 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.332550    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 22:18:05.332840 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.332644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 22:18:05.332840 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.332758    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 22:18:05.341445 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.341427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/484a6447-e587-4e1e-b292-3423d2e066cb-cert\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.341524 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.341468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb599\" (UniqueName: \"kubernetes.io/projected/484a6447-e587-4e1e-b292-3423d2e066cb-kube-api-access-wb599\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.344032 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.344016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t7wwr"]
Jul 16 22:18:05.344082 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.344044    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t5smt"]
Jul 16 22:18:05.344082 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.344061    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nrbck"]
Jul 16 22:18:05.344143 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.344115    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.346643 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.346626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4hsw2\""
Jul 16 22:18:05.346731 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.346626    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 22:18:05.346731 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.346632    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 22:18:05.362112 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.362095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrbck"]
Jul 16 22:18:05.362209 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.362196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.365676 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.365657    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 22:18:05.365676 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.365668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 22:18:05.368498 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.368479    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 22:18:05.368590 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.368553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 22:18:05.368590 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.368557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cgkc2\""
Jul 16 22:18:05.442084 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzdc\" (UniqueName: \"kubernetes.io/projected/01d1b9d5-9451-4525-a2e1-5f52d140a07a-kube-api-access-sjzdc\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.442084 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d1b9d5-9451-4525-a2e1-5f52d140a07a-tmp-dir\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-crio-socket\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6fh\" (UniqueName: \"kubernetes.io/projected/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-api-access-8x6fh\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/484a6447-e587-4e1e-b292-3423d2e066cb-cert\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01d1b9d5-9451-4525-a2e1-5f52d140a07a-config-volume\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-data-volume\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb599\" (UniqueName: \"kubernetes.io/projected/484a6447-e587-4e1e-b292-3423d2e066cb-kube-api-access-wb599\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.442318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.442305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01d1b9d5-9451-4525-a2e1-5f52d140a07a-metrics-tls\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.446463 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.446436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/484a6447-e587-4e1e-b292-3423d2e066cb-cert\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.450810 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.450788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb599\" (UniqueName: \"kubernetes.io/projected/484a6447-e587-4e1e-b292-3423d2e066cb-kube-api-access-wb599\") pod \"ingress-canary-t7wwr\" (UID: \"484a6447-e587-4e1e-b292-3423d2e066cb\") " pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.456495 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.456472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:05.456590 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.456569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:05.456648 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.456596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:05.459554 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.459538    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:18:05.459776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.459762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jqtwh\""
Jul 16 22:18:05.459911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.459893    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:18:05.460578 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.460564    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:18:05.460650 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.460564    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjst2\""
Jul 16 22:18:05.460650 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.460603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:18:05.542890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.542813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01d1b9d5-9451-4525-a2e1-5f52d140a07a-config-volume\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.542890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.542873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-data-volume\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543060 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.542896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01d1b9d5-9451-4525-a2e1-5f52d140a07a-metrics-tls\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.543060 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.542938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzdc\" (UniqueName: \"kubernetes.io/projected/01d1b9d5-9451-4525-a2e1-5f52d140a07a-kube-api-access-sjzdc\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.543060 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.542981    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d1b9d5-9451-4525-a2e1-5f52d140a07a-tmp-dir\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.543200 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-crio-socket\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543200 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6fh\" (UniqueName: \"kubernetes.io/projected/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-api-access-8x6fh\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543318 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d1b9d5-9451-4525-a2e1-5f52d140a07a-tmp-dir\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.543483 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-crio-socket\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543540 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01d1b9d5-9451-4525-a2e1-5f52d140a07a-config-volume\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.543602 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-data-volume\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.543869 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.543847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.545381 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.545364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.545435 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.545409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01d1b9d5-9451-4525-a2e1-5f52d140a07a-metrics-tls\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.552113 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.552091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6fh\" (UniqueName: \"kubernetes.io/projected/e33299f5-3377-4a51-bdb2-478c0e3d6a7d-kube-api-access-8x6fh\") pod \"insights-runtime-extractor-nrbck\" (UID: \"e33299f5-3377-4a51-bdb2-478c0e3d6a7d\") " pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.561486 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.561467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzdc\" (UniqueName: \"kubernetes.io/projected/01d1b9d5-9451-4525-a2e1-5f52d140a07a-kube-api-access-sjzdc\") pod \"dns-default-t5smt\" (UID: \"01d1b9d5-9451-4525-a2e1-5f52d140a07a\") " pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.639406 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.639382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t7wwr"
Jul 16 22:18:05.651645 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.651618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:05.670275 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.670258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrbck"
Jul 16 22:18:05.867675 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.867645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t7wwr"]
Jul 16 22:18:05.871647 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.871620    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t5smt"]
Jul 16 22:18:05.872400 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:05.872336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484a6447_e587_4e1e_b292_3423d2e066cb.slice/crio-a8a108d21ea55b3e8524d85771f9a9956fcc602eb274a89656c53bdd3b4deb7e WatchSource:0}: Error finding container a8a108d21ea55b3e8524d85771f9a9956fcc602eb274a89656c53bdd3b4deb7e: Status 404 returned error can't find the container with id a8a108d21ea55b3e8524d85771f9a9956fcc602eb274a89656c53bdd3b4deb7e
Jul 16 22:18:05.872565 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:05.872545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrbck"]
Jul 16 22:18:06.666815 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:06.666765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrbck" event={"ID":"e33299f5-3377-4a51-bdb2-478c0e3d6a7d","Type":"ContainerStarted","Data":"4be7c906ca40851b9952085771354fcb10074b86464391c3218b645fbdabfc5a"}
Jul 16 22:18:06.668600 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:06.668512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5smt" event={"ID":"01d1b9d5-9451-4525-a2e1-5f52d140a07a","Type":"ContainerStarted","Data":"551e6e65c6289f8314667eaf38d4dfedc28f89aa2cc0e2ab08b2252bf22a36f9"}
Jul 16 22:18:06.670006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:06.669982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t7wwr" event={"ID":"484a6447-e587-4e1e-b292-3423d2e066cb","Type":"ContainerStarted","Data":"a8a108d21ea55b3e8524d85771f9a9956fcc602eb274a89656c53bdd3b4deb7e"}
Jul 16 22:18:12.687862 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:12.687657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t7wwr" event={"ID":"484a6447-e587-4e1e-b292-3423d2e066cb","Type":"ContainerStarted","Data":"b78100dfe5794745402dfc10890661e23decf17ee896602c3d1738f5362a8da3"}
Jul 16 22:18:12.689602 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:12.689579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5smt" event={"ID":"01d1b9d5-9451-4525-a2e1-5f52d140a07a","Type":"ContainerStarted","Data":"8891f1840d7fa6be69ac232a2890b521384e33e3ca6007275efdd45776936c77"}
Jul 16 22:18:12.707633 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:12.707590    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t7wwr" podStartSLOduration=1.073669452 podStartE2EDuration="7.707574881s" totalImagesPullingTime="6.633905429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:12.707149753 +0000 UTC m=+54.775472809" watchObservedRunningTime="2026-07-16 22:18:12.707574881 +0000 UTC m=+54.775897940"
Jul 16 22:18:13.450857 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.450818    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-tnxm8"]
Jul 16 22:18:13.462475 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.462450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:13.465021 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.464840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 22:18:13.465021 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.464850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hww5j\""
Jul 16 22:18:13.465021 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.464889    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 22:18:13.467211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.467188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-tnxm8"]
Jul 16 22:18:13.504244 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.504217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957l4\" (UniqueName: \"kubernetes.io/projected/5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53-kube-api-access-957l4\") pod \"downloads-5bc67dcd46-tnxm8\" (UID: \"5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53\") " pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:13.605103 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.605071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957l4\" (UniqueName: \"kubernetes.io/projected/5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53-kube-api-access-957l4\") pod \"downloads-5bc67dcd46-tnxm8\" (UID: \"5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53\") " pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:13.616425 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.616231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957l4\" (UniqueName: \"kubernetes.io/projected/5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53-kube-api-access-957l4\") pod \"downloads-5bc67dcd46-tnxm8\" (UID: \"5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53\") " pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:13.694043 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.694008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrbck" event={"ID":"e33299f5-3377-4a51-bdb2-478c0e3d6a7d","Type":"ContainerStarted","Data":"2dfa42c3da6084f235162fd7ccc56a23a77ebee1c70490098e5aac75823344dd"}
Jul 16 22:18:13.696612 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.696571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t5smt" event={"ID":"01d1b9d5-9451-4525-a2e1-5f52d140a07a","Type":"ContainerStarted","Data":"d6a96cbafc13798a4d2360f25ac1bec7d359bb93da75045d44841169a31f3493"}
Jul 16 22:18:13.715062 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.714938    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t5smt" podStartSLOduration=2.086700737 podStartE2EDuration="8.714919459s" totalImagesPullingTime="6.628218722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:13.714359133 +0000 UTC m=+55.782682187" watchObservedRunningTime="2026-07-16 22:18:13.714919459 +0000 UTC m=+55.783242514"
Jul 16 22:18:13.774872 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.774842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:13.925862 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:13.925832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-tnxm8"]
Jul 16 22:18:13.929748 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:13.929593    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f375af6_8bb1_4e3c_a8ac_ad6f6c8d5e53.slice/crio-adee847d59e4789379235207cdbfe5998360432296e9c366272510caa6af0080 WatchSource:0}: Error finding container adee847d59e4789379235207cdbfe5998360432296e9c366272510caa6af0080: Status 404 returned error can't find the container with id adee847d59e4789379235207cdbfe5998360432296e9c366272510caa6af0080
Jul 16 22:18:14.700679 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:14.700636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-tnxm8" event={"ID":"5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53","Type":"ContainerStarted","Data":"adee847d59e4789379235207cdbfe5998360432296e9c366272510caa6af0080"}
Jul 16 22:18:14.701105 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:14.700972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:17.712824 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:17.712782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrbck" event={"ID":"e33299f5-3377-4a51-bdb2-478c0e3d6a7d","Type":"ContainerStarted","Data":"fd74121f3c04dbfdb0bdb47131abbb9198e1dca74a3e2f8159c84349fa459159"}
Jul 16 22:18:18.711858 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.711825    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"]
Jul 16 22:18:18.714124 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.714097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.716557 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.716530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 22:18:18.717664 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.717466    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 22:18:18.717664 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.717498    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 22:18:18.717863 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.717801    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jtnmk\""
Jul 16 22:18:18.718029 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.718012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 22:18:18.718117 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.718094    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 22:18:18.726542 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.726492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"]
Jul 16 22:18:18.749196 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.749171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.749299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.749213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496a12e6-e349-44a8-be45-867744d20e65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.749299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.749249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkbxv\" (UniqueName: \"kubernetes.io/projected/496a12e6-e349-44a8-be45-867744d20e65-kube-api-access-dkbxv\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.749299 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.749269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.850222 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.850190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.850402 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.850235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496a12e6-e349-44a8-be45-867744d20e65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.850402 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.850282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkbxv\" (UniqueName: \"kubernetes.io/projected/496a12e6-e349-44a8-be45-867744d20e65-kube-api-access-dkbxv\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.850402 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.850312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.851242 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.851196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496a12e6-e349-44a8-be45-867744d20e65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.854195 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.854143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.854306 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.854202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/496a12e6-e349-44a8-be45-867744d20e65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:18.858642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:18.858617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkbxv\" (UniqueName: \"kubernetes.io/projected/496a12e6-e349-44a8-be45-867744d20e65-kube-api-access-dkbxv\") pod \"prometheus-operator-5945b5cfd-m95pp\" (UID: \"496a12e6-e349-44a8-be45-867744d20e65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:19.026563 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:19.026476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"
Jul 16 22:18:19.639001 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:19.638970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kg9h9"
Jul 16 22:18:20.998893 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:20.998866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-m95pp"]
Jul 16 22:18:21.001310 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:21.001284    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496a12e6_e349_44a8_be45_867744d20e65.slice/crio-a50dc1da7fa8c62ff2527694b1f111422ac2932fcf032aacb05bbb1b1926ff37 WatchSource:0}: Error finding container a50dc1da7fa8c62ff2527694b1f111422ac2932fcf032aacb05bbb1b1926ff37: Status 404 returned error can't find the container with id a50dc1da7fa8c62ff2527694b1f111422ac2932fcf032aacb05bbb1b1926ff37
Jul 16 22:18:21.731686 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:21.731647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrbck" event={"ID":"e33299f5-3377-4a51-bdb2-478c0e3d6a7d","Type":"ContainerStarted","Data":"bb3112f5845fb973fb2a20c6d08a20aca5dd721a788a1798b93788ace8fd6e92"}
Jul 16 22:18:21.732903 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:21.732862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp" event={"ID":"496a12e6-e349-44a8-be45-867744d20e65","Type":"ContainerStarted","Data":"a50dc1da7fa8c62ff2527694b1f111422ac2932fcf032aacb05bbb1b1926ff37"}
Jul 16 22:18:21.749168 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:21.749123    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nrbck" podStartSLOduration=1.9212853189999999 podStartE2EDuration="16.749109998s" totalImagesPullingTime="14.827824679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:18:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:21.748553639 +0000 UTC m=+63.816876694" watchObservedRunningTime="2026-07-16 22:18:21.749109998 +0000 UTC m=+63.817433052"
Jul 16 22:18:22.168253 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.168183    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:18:22.170258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.170232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.175919 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.175868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 22:18:22.175919 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.175892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-frs6s\""
Jul 16 22:18:22.176899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.176876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 22:18:22.176899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.176918    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 22:18:22.177172 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.177153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 22:18:22.177304 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.177154    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 22:18:22.185424 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.185404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:18:22.278980 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.278942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.279149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.279056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.279149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.279112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.279149 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.279135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.279327 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.279158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.279327 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.279201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknds\" (UniqueName: \"kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.380776 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.380748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknds\" (UniqueName: \"kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.381600 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.381569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.381746 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.381618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.381746 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.381639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.383667 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.383633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.383667 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.383658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.389210 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.389174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknds\" (UniqueName: \"kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds\") pod \"console-5788b66b5-9rms6\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") " pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.482091 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.482018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:22.739666 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.739626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp" event={"ID":"496a12e6-e349-44a8-be45-867744d20e65","Type":"ContainerStarted","Data":"a4c2bbc34183ee5aaaa0100b90319ff178e87913f7b7a14d2258a85688a4cc3c"}
Jul 16 22:18:22.746310 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:22.746291    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:18:22.749239 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:22.749213    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3866fab_5832_46b3_93b2_873a5ba5607f.slice/crio-597d142d6dd264518ee07a665d90fc66993e6d0b1610e9429f359be7bb787053 WatchSource:0}: Error finding container 597d142d6dd264518ee07a665d90fc66993e6d0b1610e9429f359be7bb787053: Status 404 returned error can't find the container with id 597d142d6dd264518ee07a665d90fc66993e6d0b1610e9429f359be7bb787053
Jul 16 22:18:23.743919 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:23.743881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788b66b5-9rms6" event={"ID":"c3866fab-5832-46b3-93b2-873a5ba5607f","Type":"ContainerStarted","Data":"597d142d6dd264518ee07a665d90fc66993e6d0b1610e9429f359be7bb787053"}
Jul 16 22:18:23.745950 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:23.745924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp" event={"ID":"496a12e6-e349-44a8-be45-867744d20e65","Type":"ContainerStarted","Data":"e51983877a8b2c13e6b64fb0c4546669989cc598004ba603e895f349de7cf57d"}
Jul 16 22:18:23.761883 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:23.761839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-m95pp" podStartSLOduration=4.130627319 podStartE2EDuration="5.761825252s" totalImagesPullingTime="1.631197933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:23.761242124 +0000 UTC m=+65.829565178" watchObservedRunningTime="2026-07-16 22:18:23.761825252 +0000 UTC m=+65.830148306"
Jul 16 22:18:24.096966 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.096933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:24.099692 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.099661    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 22:18:24.110189 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.110162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53207566-66e5-4cca-909d-754cbcb13ee1-metrics-certs\") pod \"network-metrics-daemon-57ngb\" (UID: \"53207566-66e5-4cca-909d-754cbcb13ee1\") " pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:24.197500 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.197462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:24.197663 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.197555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:24.200082 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.200064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 22:18:24.200174 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.200115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 22:18:24.210086 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.210046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 22:18:24.210388 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.210370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/78b06a78-a819-426c-8aee-a671d9a09e6b-original-pull-secret\") pod \"global-pull-secret-syncer-6bwd5\" (UID: \"78b06a78-a819-426c-8aee-a671d9a09e6b\") " pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:24.221359 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.221335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6hh\" (UniqueName: \"kubernetes.io/projected/ede76fdf-2fb3-4493-8965-39daec8e3a86-kube-api-access-dt6hh\") pod \"network-check-target-nhbx5\" (UID: \"ede76fdf-2fb3-4493-8965-39daec8e3a86\") " pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:24.368105 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.368022    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jqtwh\""
Jul 16 22:18:24.372040 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.372013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6bwd5"
Jul 16 22:18:24.376056 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.376031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:24.378128 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.378101    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjst2\""
Jul 16 22:18:24.386226 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.386204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-57ngb"
Jul 16 22:18:24.539411 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.539324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6bwd5"]
Jul 16 22:18:24.707793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.707438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t5smt"
Jul 16 22:18:24.751608 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.751539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bwd5" event={"ID":"78b06a78-a819-426c-8aee-a671d9a09e6b","Type":"ContainerStarted","Data":"bde443e29ba12da97858d8b823e63ed63efc190d32cba2c0aa045afb3946c35d"}
Jul 16 22:18:24.769218 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.769184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-57ngb"]
Jul 16 22:18:24.774216 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:24.774186    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53207566_66e5_4cca_909d_754cbcb13ee1.slice/crio-4b08b3827cdaa35d65868ddc6a48713f24b9a5022b1d747c8e4785fa2070776d WatchSource:0}: Error finding container 4b08b3827cdaa35d65868ddc6a48713f24b9a5022b1d747c8e4785fa2070776d: Status 404 returned error can't find the container with id 4b08b3827cdaa35d65868ddc6a48713f24b9a5022b1d747c8e4785fa2070776d
Jul 16 22:18:24.776145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:24.775952    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nhbx5"]
Jul 16 22:18:24.784409 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:24.784371    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede76fdf_2fb3_4493_8965_39daec8e3a86.slice/crio-8b91ee99501890f52783fa25d7da04089ea71e2f5ab470b592d989548a321348 WatchSource:0}: Error finding container 8b91ee99501890f52783fa25d7da04089ea71e2f5ab470b592d989548a321348: Status 404 returned error can't find the container with id 8b91ee99501890f52783fa25d7da04089ea71e2f5ab470b592d989548a321348
Jul 16 22:18:25.759045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:25.758968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57ngb" event={"ID":"53207566-66e5-4cca-909d-754cbcb13ee1","Type":"ContainerStarted","Data":"4b08b3827cdaa35d65868ddc6a48713f24b9a5022b1d747c8e4785fa2070776d"}
Jul 16 22:18:25.760641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:25.760608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhbx5" event={"ID":"ede76fdf-2fb3-4493-8965-39daec8e3a86","Type":"ContainerStarted","Data":"8b91ee99501890f52783fa25d7da04089ea71e2f5ab470b592d989548a321348"}
Jul 16 22:18:26.099522 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.099486    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"]
Jul 16 22:18:26.104087 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.104002    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.108773 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.108750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 22:18:26.108873 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.108785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 22:18:26.108873 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.108812    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-t4xqk\""
Jul 16 22:18:26.108990 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.108954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 22:18:26.117977 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.117947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"]
Jul 16 22:18:26.126666 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.126645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hrmrz"]
Jul 16 22:18:26.129833 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.129814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.132093 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.132075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 22:18:26.133618 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.133594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 22:18:26.133943 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.133924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n649k\""
Jul 16 22:18:26.134041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.133930    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 22:18:26.215793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.215793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.216033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-textfile\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz7wl\" (UniqueName: \"kubernetes.io/projected/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-api-access-jz7wl\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.216033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-root\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.215980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-metrics-client-ca\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216033 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216006    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.216287 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-sys\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216287 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.216287 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-tls\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216287 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-wtmp\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216287 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlsx9\" (UniqueName: \"kubernetes.io/projected/92defb58-8f20-433c-a0d1-dacc3f266ff6-kube-api-access-xlsx9\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216487 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.216487 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.216487 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.216374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/505ecd5f-91bc-419a-a6e9-59bf9386366e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.317492 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-sys\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-tls\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-wtmp\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlsx9\" (UniqueName: \"kubernetes.io/projected/92defb58-8f20-433c-a0d1-dacc3f266ff6-kube-api-access-xlsx9\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.317684 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/505ecd5f-91bc-419a-a6e9-59bf9386366e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-textfile\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz7wl\" (UniqueName: \"kubernetes.io/projected/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-api-access-jz7wl\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-root\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318024 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.317908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-metrics-client-ca\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318531 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.318496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/505ecd5f-91bc-419a-a6e9-59bf9386366e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.318531 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.318514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-metrics-client-ca\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318653 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.318543    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-root\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.318653 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.318596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-sys\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.319207 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.318967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-wtmp\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.319207 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.319093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-accelerators-collector-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.319207 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.319159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-textfile\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.319451 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.319378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.319610 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.319582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.321548 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.321526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.321738 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.321701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.322110 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.322088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92defb58-8f20-433c-a0d1-dacc3f266ff6-node-exporter-tls\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.323371 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.323329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.326977 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.326954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz7wl\" (UniqueName: \"kubernetes.io/projected/505ecd5f-91bc-419a-a6e9-59bf9386366e-kube-api-access-jz7wl\") pod \"kube-state-metrics-56c5594674-lgsx5\" (UID: \"505ecd5f-91bc-419a-a6e9-59bf9386366e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.327335 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.327311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlsx9\" (UniqueName: \"kubernetes.io/projected/92defb58-8f20-433c-a0d1-dacc3f266ff6-kube-api-access-xlsx9\") pod \"node-exporter-hrmrz\" (UID: \"92defb58-8f20-433c-a0d1-dacc3f266ff6\") " pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:26.416438 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.416357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"
Jul 16 22:18:26.442052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:26.442022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hrmrz"
Jul 16 22:18:28.082421 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.082385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-f4fbcf599-v78kv"]
Jul 16 22:18:28.086324 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.086293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.090055 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 22:18:28.090055 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 22:18:28.090496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 22:18:28.090496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090040    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 22:18:28.090496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8jr7c\""
Jul 16 22:18:28.090496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090282    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 22:18:28.090496 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.090359    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ebr1g1akg7ja7\""
Jul 16 22:18:28.097979 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.097960    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f4fbcf599-v78kv"]
Jul 16 22:18:28.233594 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-grpc-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d55t\" (UniqueName: \"kubernetes.io/projected/5e0f7999-0614-4373-a481-0e5f8ecedddb-kube-api-access-7d55t\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.233790 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0f7999-0614-4373-a481-0e5f8ecedddb-metrics-client-ca\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.234041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.233824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335077 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.334999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d55t\" (UniqueName: \"kubernetes.io/projected/5e0f7999-0614-4373-a481-0e5f8ecedddb-kube-api-access-7d55t\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335077 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335077 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0f7999-0614-4373-a481-0e5f8ecedddb-metrics-client-ca\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335356 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335356 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-grpc-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335356 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335356 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.335356 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.335254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.336170 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.336119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0f7999-0614-4373-a481-0e5f8ecedddb-metrics-client-ca\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.338333 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.338455 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.338816 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.338925 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-grpc-tls\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.339016 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.339063 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.338995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5e0f7999-0614-4373-a481-0e5f8ecedddb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.343766 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.343747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d55t\" (UniqueName: \"kubernetes.io/projected/5e0f7999-0614-4373-a481-0e5f8ecedddb-kube-api-access-7d55t\") pod \"thanos-querier-f4fbcf599-v78kv\" (UID: \"5e0f7999-0614-4373-a481-0e5f8ecedddb\") " pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:28.398610 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:28.398585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:30.520658 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.520628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-86cdffd95b-ffbc4"]
Jul 16 22:18:30.530063 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.530039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.533642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.533618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8ell6ph5lccln\""
Jul 16 22:18:30.533775 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.533652    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 22:18:30.533775 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.533680    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 22:18:30.534206 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.534063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5xr5c\""
Jul 16 22:18:30.534206 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.534072    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 22:18:30.534206 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.534134    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 22:18:30.535817 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.535793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86cdffd95b-ffbc4"]
Jul 16 22:18:30.654067 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-metrics-server-audit-profiles\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654240 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-client-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654240 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-tls\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654240 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-client-certs\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654240 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654441 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-audit-log\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.654441 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.654326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpcqq\" (UniqueName: \"kubernetes.io/projected/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-kube-api-access-jpcqq\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755022 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.754989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-metrics-server-audit-profiles\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755539 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-client-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755677 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-tls\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755677 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-client-certs\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755813 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755813 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-audit-log\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.755926 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.755847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpcqq\" (UniqueName: \"kubernetes.io/projected/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-kube-api-access-jpcqq\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.757666 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.756743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.757666 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.756801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-metrics-server-audit-profiles\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.757666 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.756844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-audit-log\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.761861 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.761605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-client-ca-bundle\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.761861 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.761788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-tls\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.762038 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.761910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-secret-metrics-server-client-certs\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.767648 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.767607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpcqq\" (UniqueName: \"kubernetes.io/projected/85f6e921-1f22-4dfb-b4d1-b94bc8fe519c-kube-api-access-jpcqq\") pod \"metrics-server-86cdffd95b-ffbc4\" (UID: \"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c\") " pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:30.842979 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:30.842946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:32.304387 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.304352    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:18:32.361174 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.361135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:18:32.361309 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.361251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.364989 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.364963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 22:18:32.365417 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.364980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 22:18:32.365417 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.364980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 22:18:32.365417 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 22:18:32.365417 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 22:18:32.365417 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 22:18:32.365980 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 22:18:32.365980 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rrmk5\""
Jul 16 22:18:32.366152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 22:18:32.366152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.365947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 22:18:32.366152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366007    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cupud79ci4ked\""
Jul 16 22:18:32.366152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366000    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 22:18:32.366326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 22:18:32.366326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 22:18:32.366326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.366209    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 22:18:32.369029 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.369008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 22:18:32.471267 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471733 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471733 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471733 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471733 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59zj\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.471928 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.472168 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.472168 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.471977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.472168 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.472000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572656 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572866 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572866 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572866 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59zj\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572866 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.572866 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.572966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573177 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573616 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.573890 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.573699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.574265 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.574235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.575641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.574742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.575641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.574875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.575641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.575425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.575880 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.575843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.575934 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.575889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.576792 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.576754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.581589 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.581557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.582428 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.582403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.582555 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.582536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.582694 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.582669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.583655 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.583198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.583655 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.583302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.583655 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.583593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.583655 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.583624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.586970 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.584649    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59zj\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.586970 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.585089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.587131 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.586986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:32.673265 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:32.673112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:18:34.187357 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:34.187309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:18:36.951085 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:36.951022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92defb58_8f20_433c_a0d1_dacc3f266ff6.slice/crio-cc71e110a273a6c476f9bb8de7b39f6f527cb9edf7af69d466917cd0ad613a7a WatchSource:0}: Error finding container cc71e110a273a6c476f9bb8de7b39f6f527cb9edf7af69d466917cd0ad613a7a: Status 404 returned error can't find the container with id cc71e110a273a6c476f9bb8de7b39f6f527cb9edf7af69d466917cd0ad613a7a
Jul 16 22:18:37.110985 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.110779    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lgsx5"]
Jul 16 22:18:37.326070 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.326028    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-86cdffd95b-ffbc4"]
Jul 16 22:18:37.331200 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.331175    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-f4fbcf599-v78kv"]
Jul 16 22:18:37.390206 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:37.390164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505ecd5f_91bc_419a_a6e9_59bf9386366e.slice/crio-4d0a643173c44415a941ccdcdfc823edd85605b575fc431d23e60f6f36ec9683 WatchSource:0}: Error finding container 4d0a643173c44415a941ccdcdfc823edd85605b575fc431d23e60f6f36ec9683: Status 404 returned error can't find the container with id 4d0a643173c44415a941ccdcdfc823edd85605b575fc431d23e60f6f36ec9683
Jul 16 22:18:37.393003 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:37.392486    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6e921_1f22_4dfb_b4d1_b94bc8fe519c.slice/crio-8aceec731e42d8f7a2adf06f227ada736825ca36e7ce91482792705d087f09e0 WatchSource:0}: Error finding container 8aceec731e42d8f7a2adf06f227ada736825ca36e7ce91482792705d087f09e0: Status 404 returned error can't find the container with id 8aceec731e42d8f7a2adf06f227ada736825ca36e7ce91482792705d087f09e0
Jul 16 22:18:37.393003 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:37.392696    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0f7999_0614_4373_a481_0e5f8ecedddb.slice/crio-428e06fd502b4bcf06d87ce69726c2830a7130332cb82c4cda814ea986dc3374 WatchSource:0}: Error finding container 428e06fd502b4bcf06d87ce69726c2830a7130332cb82c4cda814ea986dc3374: Status 404 returned error can't find the container with id 428e06fd502b4bcf06d87ce69726c2830a7130332cb82c4cda814ea986dc3374
Jul 16 22:18:37.519264 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.519237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:18:37.521877 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:18:37.521829    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb527a6_4583_4c72_b10e_886cda3a4685.slice/crio-9ad07d3f62eda7d39f8f7bb4b168572d2e86c25fb4d27b7a3c5363648bfb0722 WatchSource:0}: Error finding container 9ad07d3f62eda7d39f8f7bb4b168572d2e86c25fb4d27b7a3c5363648bfb0722: Status 404 returned error can't find the container with id 9ad07d3f62eda7d39f8f7bb4b168572d2e86c25fb4d27b7a3c5363648bfb0722
Jul 16 22:18:37.809531 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.809500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-tnxm8" event={"ID":"5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53","Type":"ContainerStarted","Data":"6968376bdd194164774d6ada1217722f863f0b2b253cf744346e4e89df8a2380"}
Jul 16 22:18:37.810204 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.810154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:37.811785 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.811755    2579 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-tnxm8 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" start-of-body=
Jul 16 22:18:37.811879 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.811812    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-tnxm8" podUID="5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused"
Jul 16 22:18:37.813807 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.813778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788b66b5-9rms6" event={"ID":"c3866fab-5832-46b3-93b2-873a5ba5607f","Type":"ContainerStarted","Data":"6ebb2bb18180740aed9a7b12ef01cccfd67bea43fca03c031d3dc5e18d52eac4"}
Jul 16 22:18:37.816185 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.816129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57ngb" event={"ID":"53207566-66e5-4cca-909d-754cbcb13ee1","Type":"ContainerStarted","Data":"33e5213f7aaedacb0f2c14f3ac2770f46e7a1a786763585a4baa4ad6852f5fb3"}
Jul 16 22:18:37.817961 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.817904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nhbx5" event={"ID":"ede76fdf-2fb3-4493-8965-39daec8e3a86","Type":"ContainerStarted","Data":"4b44ce041ab41a0f8e7e7a90aee152cb5e1277831370fec4897b29d7322e56a1"}
Jul 16 22:18:37.818079 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.817999    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:18:37.819424 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.819401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4" event={"ID":"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c","Type":"ContainerStarted","Data":"8aceec731e42d8f7a2adf06f227ada736825ca36e7ce91482792705d087f09e0"}
Jul 16 22:18:37.822578 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.822554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"428e06fd502b4bcf06d87ce69726c2830a7130332cb82c4cda814ea986dc3374"}
Jul 16 22:18:37.824322 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.824288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6bwd5" event={"ID":"78b06a78-a819-426c-8aee-a671d9a09e6b","Type":"ContainerStarted","Data":"1de4b8287c9cfd55c47124c859157234345f8a701d09cba8abc22d7fb8c0b1b4"}
Jul 16 22:18:37.826145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.826077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"9ad07d3f62eda7d39f8f7bb4b168572d2e86c25fb4d27b7a3c5363648bfb0722"}
Jul 16 22:18:37.828462 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.828427    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-tnxm8" podStartSLOduration=2.767199056 podStartE2EDuration="24.828414205s" totalImagesPullingTime="22.061215149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:37.826757499 +0000 UTC m=+79.895080554" watchObservedRunningTime="2026-07-16 22:18:37.828414205 +0000 UTC m=+79.896737251"
Jul 16 22:18:37.828670 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.828487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5" event={"ID":"505ecd5f-91bc-419a-a6e9-59bf9386366e","Type":"ContainerStarted","Data":"4d0a643173c44415a941ccdcdfc823edd85605b575fc431d23e60f6f36ec9683"}
Jul 16 22:18:37.830059 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.830036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrmrz" event={"ID":"92defb58-8f20-433c-a0d1-dacc3f266ff6","Type":"ContainerStarted","Data":"cc71e110a273a6c476f9bb8de7b39f6f527cb9edf7af69d466917cd0ad613a7a"}
Jul 16 22:18:37.847202 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.847037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5788b66b5-9rms6" podStartSLOduration=1.647069178 podStartE2EDuration="15.84701285s" totalImagesPullingTime="14.199943672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:37.845676951 +0000 UTC m=+79.914000007" watchObservedRunningTime="2026-07-16 22:18:37.84701285 +0000 UTC m=+79.915335914"
Jul 16 22:18:37.863777 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.863696    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nhbx5" podStartSLOduration=67.140139652 podStartE2EDuration="1m19.863674611s" totalImagesPullingTime="12.723534959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:37.862797501 +0000 UTC m=+79.931120558" watchObservedRunningTime="2026-07-16 22:18:37.863674611 +0000 UTC m=+79.931997666"
Jul 16 22:18:37.880526 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:37.880432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6bwd5" podStartSLOduration=67.475930988 podStartE2EDuration="1m19.880411541s" totalImagesPullingTime="12.404480553s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:37.879900508 +0000 UTC m=+79.948223565" watchObservedRunningTime="2026-07-16 22:18:37.880411541 +0000 UTC m=+79.948734598"
Jul 16 22:18:38.840175 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:38.839434    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-57ngb" event={"ID":"53207566-66e5-4cca-909d-754cbcb13ee1","Type":"ContainerStarted","Data":"dc4500657e0e434ffd2de9fd8a512decaf6d7d5eb772006bbf8fe5b257443ad8"}
Jul 16 22:18:38.856272 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:38.856214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-57ngb" podStartSLOduration=68.119692763 podStartE2EDuration="1m20.856184147s" totalImagesPullingTime="12.736491384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:38.85427324 +0000 UTC m=+80.922596296" watchObservedRunningTime="2026-07-16 22:18:38.856184147 +0000 UTC m=+80.924507203"
Jul 16 22:18:38.856740 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:38.856432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-tnxm8"
Jul 16 22:18:42.482136 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.482103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:18:42.859355 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.859313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4" event={"ID":"85f6e921-1f22-4dfb-b4d1-b94bc8fe519c","Type":"ContainerStarted","Data":"97ab48205a3433e2143eed0167bf9f68f56636906a9c54aae012c1d15d41eb95"}
Jul 16 22:18:42.862474 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.862440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"321662616b17983bde8915d4723c8b7762f77656ff4dedbea564a4c8579c9581"}
Jul 16 22:18:42.862628 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.862535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"b9eecd91046e1797ad4088c1ba4f15e050a9c9b08eccf5e56fbf3321d170cc22"}
Jul 16 22:18:42.862628 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.862583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"627c379e24846d0bebad5ce9e61dc956421558606511809da4485847e5909318"}
Jul 16 22:18:42.864338 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.864318    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa" exitCode=0
Jul 16 22:18:42.864973 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.864491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"}
Jul 16 22:18:42.868357 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.868330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5" event={"ID":"505ecd5f-91bc-419a-a6e9-59bf9386366e","Type":"ContainerStarted","Data":"061757d1b81f709a148b7b4adfb5456a32a8366ccfe159eb3b3f2c59891c39fe"}
Jul 16 22:18:42.868459 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.868386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5" event={"ID":"505ecd5f-91bc-419a-a6e9-59bf9386366e","Type":"ContainerStarted","Data":"430ceb119d3cb0b8059f6a4361470f00344cb37f683644b51e579db6e5396edf"}
Jul 16 22:18:42.868459 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.868413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5" event={"ID":"505ecd5f-91bc-419a-a6e9-59bf9386366e","Type":"ContainerStarted","Data":"55dba5cd30a08769f13c1a65b5913a0102338c0e25223ede7f4a915f9275fdc5"}
Jul 16 22:18:42.871480 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.870823    2579 generic.go:356] "Generic (PLEG): container finished" podID="92defb58-8f20-433c-a0d1-dacc3f266ff6" containerID="9b4d25094155013744f306c08304639ea5207e9b9e90b6df51fa3b34c75f6843" exitCode=0
Jul 16 22:18:42.871480 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.870905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrmrz" event={"ID":"92defb58-8f20-433c-a0d1-dacc3f266ff6","Type":"ContainerDied","Data":"9b4d25094155013744f306c08304639ea5207e9b9e90b6df51fa3b34c75f6843"}
Jul 16 22:18:42.884551 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.884484    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4" podStartSLOduration=8.395315235 podStartE2EDuration="12.884464527s" totalImagesPullingTime="4.489149292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:42.883141227 +0000 UTC m=+84.951464295" watchObservedRunningTime="2026-07-16 22:18:42.884464527 +0000 UTC m=+84.952787582"
Jul 16 22:18:42.905974 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:42.905910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-lgsx5" podStartSLOduration=12.061918353 podStartE2EDuration="16.905889632s" totalImagesPullingTime="4.843971279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:42.904619912 +0000 UTC m=+84.972942968" watchObservedRunningTime="2026-07-16 22:18:42.905889632 +0000 UTC m=+84.974212687"
Jul 16 22:18:43.890070 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:43.889225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrmrz" event={"ID":"92defb58-8f20-433c-a0d1-dacc3f266ff6","Type":"ContainerStarted","Data":"fb92afd561d414187992a14a7dda2dff611492e666ddf2459dcc80351262411d"}
Jul 16 22:18:43.890070 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:43.889396    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hrmrz" event={"ID":"92defb58-8f20-433c-a0d1-dacc3f266ff6","Type":"ContainerStarted","Data":"2e314522fb02894f3d79bf8be09ea8cd0d9d960e8f960ef7dff6c3d442a5b4cd"}
Jul 16 22:18:43.913004 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:43.912839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hrmrz" podStartSLOduration=12.779785384 podStartE2EDuration="17.912786959s" totalImagesPullingTime="5.029943435s" totalInitContainerRuntime="103.05814ms" isStatefulPod=true podCreationTimestamp="2026-07-16 22:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:43.911066992 +0000 UTC m=+85.979390073" watchObservedRunningTime="2026-07-16 22:18:43.912786959 +0000 UTC m=+85.981110013"
Jul 16 22:18:44.901845 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:44.901800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"a8e62f92afb04b6c0156de360e994201ea9b4a8d11773895746a7d4f5a5c7ef1"}
Jul 16 22:18:44.902388 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:44.901887    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:44.902388 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:44.901928    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"513e9ab3fe6afcf549f9310f710e5e0ddc3e2718c618486986e006e5cce4d8dd"}
Jul 16 22:18:44.902388 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:44.901965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" event={"ID":"5e0f7999-0614-4373-a481-0e5f8ecedddb","Type":"ContainerStarted","Data":"4d18c56fb0cd25a3eadae94dd60b8d0aca8470d627d77137eaa55e5bf32c7aa8"}
Jul 16 22:18:44.931433 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:44.931374    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv" podStartSLOduration=10.860026486 podStartE2EDuration="16.93135116s" totalImagesPullingTime="6.071324674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:44.930181225 +0000 UTC m=+86.998504280" watchObservedRunningTime="2026-07-16 22:18:44.93135116 +0000 UTC m=+86.999674215"
Jul 16 22:18:47.917123 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:47.917085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"}
Jul 16 22:18:47.917474 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:47.917160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"}
Jul 16 22:18:48.924411 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:48.924377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"}
Jul 16 22:18:48.924899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:48.924446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"}
Jul 16 22:18:48.924899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:48.924471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"}
Jul 16 22:18:48.924899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:48.924495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerStarted","Data":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"}
Jul 16 22:18:48.956634 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:48.956561    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.804860291 podStartE2EDuration="16.956541071s" totalImagesPullingTime="9.130477748s" totalInitContainerRuntime="21.203032ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:48.953961088 +0000 UTC m=+91.022284142" watchObservedRunningTime="2026-07-16 22:18:48.956541071 +0000 UTC m=+91.024864138"
Jul 16 22:18:50.843442 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:50.843404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:50.843875 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:50.843468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:18:50.912551 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:50.912526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-f4fbcf599-v78kv"
Jul 16 22:18:52.673354 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:18:52.673307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:02.849688 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:02.849649    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5788b66b5-9rms6" podUID="c3866fab-5832-46b3-93b2-873a5ba5607f" containerName="console" containerID="cri-o://6ebb2bb18180740aed9a7b12ef01cccfd67bea43fca03c031d3dc5e18d52eac4" gracePeriod=15
Jul 16 22:19:02.978249 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:02.978219    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3866fab-5832-46b3-93b2-873a5ba5607f" containerID="6ebb2bb18180740aed9a7b12ef01cccfd67bea43fca03c031d3dc5e18d52eac4" exitCode=0
Jul 16 22:19:02.978406 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:02.978291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788b66b5-9rms6" event={"ID":"c3866fab-5832-46b3-93b2-873a5ba5607f","Type":"ContainerDied","Data":"6ebb2bb18180740aed9a7b12ef01cccfd67bea43fca03c031d3dc5e18d52eac4"}
Jul 16 22:19:03.126854 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.126783    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:19:03.156985 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.156949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sknds\" (UniqueName: \"kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157107    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config\") pod \"c3866fab-5832-46b3-93b2-873a5ba5607f\" (UID: \"c3866fab-5832-46b3-93b2-873a5ba5607f\") "
Jul 16 22:19:03.157510 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config" (OuterVolumeSpecName: "console-config") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:03.157580 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.157551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:03.162732 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.158187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:03.162732 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.160206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds" (OuterVolumeSpecName: "kube-api-access-sknds") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "kube-api-access-sknds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:03.164408 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.164367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:03.165317 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.165162    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3866fab-5832-46b3-93b2-873a5ba5607f" (UID: "c3866fab-5832-46b3-93b2-873a5ba5607f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:03.258470 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258433    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-console-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.258470 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258462    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.258470 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258473    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.258740 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258486    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3866fab-5832-46b3-93b2-873a5ba5607f-oauth-serving-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.258740 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258494    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3866fab-5832-46b3-93b2-873a5ba5607f-console-oauth-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.258740 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.258504    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sknds\" (UniqueName: \"kubernetes.io/projected/c3866fab-5832-46b3-93b2-873a5ba5607f-kube-api-access-sknds\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:03.983050 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.983017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5788b66b5-9rms6" event={"ID":"c3866fab-5832-46b3-93b2-873a5ba5607f","Type":"ContainerDied","Data":"597d142d6dd264518ee07a665d90fc66993e6d0b1610e9429f359be7bb787053"}
Jul 16 22:19:03.983050 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.983040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5788b66b5-9rms6"
Jul 16 22:19:03.983534 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:03.983075    2579 scope.go:117] "RemoveContainer" containerID="6ebb2bb18180740aed9a7b12ef01cccfd67bea43fca03c031d3dc5e18d52eac4"
Jul 16 22:19:04.002935 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:04.002905    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:19:04.006787 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:04.006765    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5788b66b5-9rms6"]
Jul 16 22:19:04.461311 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:04.461278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3866fab-5832-46b3-93b2-873a5ba5607f" path="/var/lib/kubelet/pods/c3866fab-5832-46b3-93b2-873a5ba5607f/volumes"
Jul 16 22:19:08.841491 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:08.841458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nhbx5"
Jul 16 22:19:10.848439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:10.848411    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:19:10.852290 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:10.852260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-86cdffd95b-ffbc4"
Jul 16 22:19:32.674213 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:32.674168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:32.738260 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:32.738233    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:33.102647 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:33.102624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:50.719037 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719000    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719272    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="prometheus" containerID="cri-o://d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83" gracePeriod=600
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719320    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy" containerID="cri-o://ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d" gracePeriod=600
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719347    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="thanos-sidecar" containerID="cri-o://8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7" gracePeriod=600
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719374    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1" gracePeriod=600
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719374    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-web" containerID="cri-o://37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98" gracePeriod=600
Jul 16 22:19:50.719564 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.719493    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="config-reloader" containerID="cri-o://b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b" gracePeriod=600
Jul 16 22:19:50.975963 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:50.975903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.045346 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045346 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045357    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045425    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c59zj\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045519    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045544    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.045601 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045609    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045667    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045834    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046045 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.045891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets\") pod \"efb527a6-4583-4c72-b10e-886cda3a4685\" (UID: \"efb527a6-4583-4c72-b10e-886cda3a4685\") "
Jul 16 22:19:51.046498 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.046124    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.046498 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.046318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.046664 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.046634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.047252 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.047227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.048277 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.048249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.048663 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.048639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:51.048858 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.048663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config" (OuterVolumeSpecName: "config") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.049082 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.049055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.049392 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.049283    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.049485 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.049443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj" (OuterVolumeSpecName: "kube-api-access-c59zj") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "kube-api-access-c59zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:19:51.049999 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.049880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.049999 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.049897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.050320 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.050225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.050571 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.050546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:51.050734 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.050658    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:51.050810 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.050757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.051026 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.050995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.051768 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.051749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.052685 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.052667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out" (OuterVolumeSpecName: "config-out") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:19:51.060671 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.060644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config" (OuterVolumeSpecName: "web-config") pod "efb527a6-4583-4c72-b10e-886cda3a4685" (UID: "efb527a6-4583-4c72-b10e-886cda3a4685"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:19:51.146694 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146644    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146694 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146679    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146694 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146693    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-web-config\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146724    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-metrics-client-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146735    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-kube-rbac-proxy\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146743    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146752    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-db\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146761    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146770    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146778    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/efb527a6-4583-4c72-b10e-886cda3a4685-config-out\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146786    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146794    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-tls-assets\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146803    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-grpc-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146811    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-metrics-client-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146819    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146829    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/efb527a6-4583-4c72-b10e-886cda3a4685-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146837    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146847    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c59zj\" (UniqueName: \"kubernetes.io/projected/efb527a6-4583-4c72-b10e-886cda3a4685-kube-api-access-c59zj\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.146949 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.146855    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efb527a6-4583-4c72-b10e-886cda3a4685-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:19:51.152596 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152570    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1" exitCode=0
Jul 16 22:19:51.152596 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152593    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d" exitCode=0
Jul 16 22:19:51.152596 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152599    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98" exitCode=0
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152606    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7" exitCode=0
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152613    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b" exitCode=0
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152621    2579 generic.go:356] "Generic (PLEG): container finished" podID="efb527a6-4583-4c72-b10e-886cda3a4685" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83" exitCode=0
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"}
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152645    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"}
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152734    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"}
Jul 16 22:19:51.152814 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"}
Jul 16 22:19:51.153125 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"}
Jul 16 22:19:51.153125 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"}
Jul 16 22:19:51.153125 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.152924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"efb527a6-4583-4c72-b10e-886cda3a4685","Type":"ContainerDied","Data":"9ad07d3f62eda7d39f8f7bb4b168572d2e86c25fb4d27b7a3c5363648bfb0722"}
Jul 16 22:19:51.160476 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.160385    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.167523 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.167504    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.173629 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.173613    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.176767 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.176743    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:51.182262 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.182210    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:51.185113 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.185090    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.191803 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.191775    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.198669 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.198549    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.204553 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.204538    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.204823 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.204802    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.204888 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.204832    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.204888 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.204870    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.205126 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.205109    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.205179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205131    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.205179 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205146    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.205388 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.205369    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.205477 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205390    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.205477 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205404    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.205634 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.205617    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.205673 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205640    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.205673 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205654    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.205901 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.205887    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.205943 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205905    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.205943 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.205919    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.206128 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.206107    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.206165 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206133    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.206165 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206148    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.206376 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:19:51.206359    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.206439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206392    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.206439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206413    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.206642 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.206806 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206644    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.206876 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206862    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.206934 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.206877    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.207122 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207105    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.207122 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207123    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.207309 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207294    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.207357 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207309    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.207662 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207532    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.207662 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207557    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.207923 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207878    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.207923 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.207907    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.208621 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.208588    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.208726 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.208622    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.208911 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.208892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.208981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.208935    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.209178 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209159    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.209178 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209177    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.209339 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209323    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:51.209400 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209383    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.209439 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209402    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.209582 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209567    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.209630 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209583    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.209630 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="config-reloader"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209634    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="config-reloader"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-web"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-web"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209662    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209673    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3866fab-5832-46b3-93b2-873a5ba5607f" containerName="console"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209682    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3866fab-5832-46b3-93b2-873a5ba5607f" containerName="console"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209701    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="prometheus"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209722    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="prometheus"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-thanos"
Jul 16 22:19:51.209730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209735    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-thanos"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209748    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="init-config-reloader"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209757    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="init-config-reloader"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209766    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="thanos-sidecar"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209773    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="thanos-sidecar"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209815    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="config-reloader"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-web"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy-thanos"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209762    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209848    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209833    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="prometheus"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209874    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="kube-rbac-proxy"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209886    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" containerName="thanos-sidecar"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.209895    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3866fab-5832-46b3-93b2-873a5ba5607f" containerName="console"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210077    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.210098 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210101    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.210622 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210275    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.210622 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210288    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.210622 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210504    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.210622 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210522    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.210804 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.210841 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.210804    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.211041 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.211086 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211042    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.211260 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211241    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.211260 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211260    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.211480 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.211521 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211481    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.211676 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211655    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.211781 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211682    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.211930 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211913    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.211978 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.211930    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.212119 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212104    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.212119 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212119    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.212284 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212265    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.212347 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212285    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.212471 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212455    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.212520 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212471    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.212672 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212652    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.212730 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212673    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.212881 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212863    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.212941 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.212883    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.213099 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213083    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.213144 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213100    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.213297 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213278    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.213372 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213298    2579 scope.go:117] "RemoveContainer" containerID="c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"
Jul 16 22:19:51.213662 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.213765 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213739    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1"} err="failed to get container status \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": rpc error: code = NotFound desc = could not find container \"c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1\": container with ID starting with c7aabd9f4ad2796101f0a99da5db4ac790b61bcb06abf3f69e9aae1ea1c358a1 not found: ID does not exist"
Jul 16 22:19:51.213765 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.213764    2579 scope.go:117] "RemoveContainer" containerID="ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"
Jul 16 22:19:51.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214083    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d"} err="failed to get container status \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": rpc error: code = NotFound desc = could not find container \"ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d\": container with ID starting with ee12c91d760edf24e753dc780caebb3f30684334dc09b36194032d05ab56743d not found: ID does not exist"
Jul 16 22:19:51.214348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214129    2579 scope.go:117] "RemoveContainer" containerID="37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"
Jul 16 22:19:51.214500 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214486    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98"} err="failed to get container status \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": rpc error: code = NotFound desc = could not find container \"37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98\": container with ID starting with 37bc7c4b5ec371989bf35839de90a0fbfc7023dc7df1c16c426992eaf45a9e98 not found: ID does not exist"
Jul 16 22:19:51.214568 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214503    2579 scope.go:117] "RemoveContainer" containerID="8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"
Jul 16 22:19:51.214761 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214734    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7"} err="failed to get container status \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": rpc error: code = NotFound desc = could not find container \"8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7\": container with ID starting with 8b4fc1ae4a969e1ad76e7934fb828028937efe2f6f98ab429aa907e7e42376b7 not found: ID does not exist"
Jul 16 22:19:51.214761 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214760    2579 scope.go:117] "RemoveContainer" containerID="b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"
Jul 16 22:19:51.215015 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.214995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b"} err="failed to get container status \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": rpc error: code = NotFound desc = could not find container \"b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b\": container with ID starting with b2f84c3bf724180ddce1392e42ab0fa3ff7c7631468a466cc987f99d58cf5f9b not found: ID does not exist"
Jul 16 22:19:51.215085 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.215017    2579 scope.go:117] "RemoveContainer" containerID="d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"
Jul 16 22:19:51.215232 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.215215    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83"} err="failed to get container status \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": rpc error: code = NotFound desc = could not find container \"d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83\": container with ID starting with d756cd72e49d5d887bd28ca557b37b1bfd59830354b4cbce228ed609944bad83 not found: ID does not exist"
Jul 16 22:19:51.215292 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.215233    2579 scope.go:117] "RemoveContainer" containerID="62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"
Jul 16 22:19:51.215410 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.215395    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa"} err="failed to get container status \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": rpc error: code = NotFound desc = could not find container \"62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa\": container with ID starting with 62b227cfdb501b11f5b0f08e35db1101a9bb46866a67ff947bd26ae770bac4fa not found: ID does not exist"
Jul 16 22:19:51.216701 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216685    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 16 22:19:51.216812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 16 22:19:51.216812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 16 22:19:51.216812 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216774    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216833    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216909    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 16 22:19:51.216959 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.216956    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 16 22:19:51.217254 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.217165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rrmk5\""
Jul 16 22:19:51.217326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.217312    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 16 22:19:51.217326 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.217321    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 16 22:19:51.217408 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.217321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cupud79ci4ked\""
Jul 16 22:19:51.220284 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.220267    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 16 22:19:51.223859 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.223843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 16 22:19:51.234865 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.234812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:51.247918 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.247901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.247927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv7bz\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-kube-api-access-rv7bz\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.247948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.247971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248006 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.247994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248224 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.248694 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.248481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349673 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv7bz\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-kube-api-access-rv7bz\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349673 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349921 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349921 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349921 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.349921 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.349986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.350518 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.350392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.351925 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.351610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.352116 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.352091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.353791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.353280    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.353791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.353343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.353791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.353481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.353791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.353671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.353791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.353733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354108 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354108 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354383 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d75575e-b323-40ab-8e75-0d4177783b20-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354490 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354490 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.354685 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.354666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d75575e-b323-40ab-8e75-0d4177783b20-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.355081 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.355062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.355125 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.355081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.355454 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.355435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.355768 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.355752    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.356114 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.356093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d75575e-b323-40ab-8e75-0d4177783b20-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.358471 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.358448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv7bz\" (UniqueName: \"kubernetes.io/projected/9d75575e-b323-40ab-8e75-0d4177783b20-kube-api-access-rv7bz\") pod \"prometheus-k8s-0\" (UID: \"9d75575e-b323-40ab-8e75-0d4177783b20\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.524359 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.524274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:19:51.661330 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:51.661307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 16 22:19:51.662906 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:19:51.662879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d75575e_b323_40ab_8e75_0d4177783b20.slice/crio-e96c5c461b382f7f23790645d38483c94c2c59001421910432377d0bca87f789 WatchSource:0}: Error finding container e96c5c461b382f7f23790645d38483c94c2c59001421910432377d0bca87f789: Status 404 returned error can't find the container with id e96c5c461b382f7f23790645d38483c94c2c59001421910432377d0bca87f789
Jul 16 22:19:52.158425 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:52.158383    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d75575e-b323-40ab-8e75-0d4177783b20" containerID="bd5291f5092f3d8eececbe583718f37301b1d342841d2c05ab1a3b68614f9cc8" exitCode=0
Jul 16 22:19:52.158425 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:52.158430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerDied","Data":"bd5291f5092f3d8eececbe583718f37301b1d342841d2c05ab1a3b68614f9cc8"}
Jul 16 22:19:52.158862 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:52.158483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"e96c5c461b382f7f23790645d38483c94c2c59001421910432377d0bca87f789"}
Jul 16 22:19:52.463348 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:52.463320    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb527a6-4583-4c72-b10e-886cda3a4685" path="/var/lib/kubelet/pods/efb527a6-4583-4c72-b10e-886cda3a4685/volumes"
Jul 16 22:19:53.166816 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.166778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"a810b2375455001010455b1019b33c393b3022d08e2fdb137b47175f48e8a4f7"}
Jul 16 22:19:53.167156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.166858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"ed38edabfb4c752a21924291d254f9bf9476e93cbc979789e4d53b5b42207fa7"}
Jul 16 22:19:53.167156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.166901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"5f680588a92c0703499cf56dafc70a61e213164c0f4069d90e1d1b1fa9e01350"}
Jul 16 22:19:53.167156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.166944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"68fd69df7e863dd42afe24875f94eeec340fe32a7a8f8e6f3e92cb0ca91f549c"}
Jul 16 22:19:53.167156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.166980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"2c94797f2cdecd76c353ed4635eb430057f351d01e660b37cc398ed2b77e6c85"}
Jul 16 22:19:53.167156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.167017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d75575e-b323-40ab-8e75-0d4177783b20","Type":"ContainerStarted","Data":"10270c0fe1d04fcb5980a3757958d48ba08283856f49671139da3b060d9466e6"}
Jul 16 22:19:53.200307 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:53.200251    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.179270875 podStartE2EDuration="2.20023344s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.962565ms" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:53.198123937 +0000 UTC m=+155.266447013" watchObservedRunningTime="2026-07-16 22:19:53.20023344 +0000 UTC m=+155.268556495"
Jul 16 22:19:56.524745 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:19:56.524690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:20:51.524445 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:20:51.524413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:20:51.539870 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:20:51.539838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:20:52.398150 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:20:52.398116    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 16 22:21:03.753947 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.753913    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dlt79"]
Jul 16 22:21:03.756077 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.756052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.760557 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.760531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 22:21:03.760557 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.760554    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 22:21:03.761285 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.761258    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lwjlv\""
Jul 16 22:21:03.770332 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.770310    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dlt79"]
Jul 16 22:21:03.836255 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.836225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-bound-sa-token\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.836419 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.836303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsdqd\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-kube-api-access-rsdqd\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.937539 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.937505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-bound-sa-token\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.937731 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.937601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsdqd\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-kube-api-access-rsdqd\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.945828 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.945796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-bound-sa-token\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:03.945973 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:03.945868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsdqd\" (UniqueName: \"kubernetes.io/projected/971b2522-0c80-417c-9228-4611be3232c9-kube-api-access-rsdqd\") pod \"cert-manager-55b4d954d5-dlt79\" (UID: \"971b2522-0c80-417c-9228-4611be3232c9\") " pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:04.072404 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:04.072375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dlt79"
Jul 16 22:21:04.191491 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:04.191467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dlt79"]
Jul 16 22:21:04.193789 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:21:04.193765    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971b2522_0c80_417c_9228_4611be3232c9.slice/crio-ec17d97822f424fab7e692b53bb40ea86fc1f48a72de876f8391f1ca54d548f7 WatchSource:0}: Error finding container ec17d97822f424fab7e692b53bb40ea86fc1f48a72de876f8391f1ca54d548f7: Status 404 returned error can't find the container with id ec17d97822f424fab7e692b53bb40ea86fc1f48a72de876f8391f1ca54d548f7
Jul 16 22:21:04.423401 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:04.423324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dlt79" event={"ID":"971b2522-0c80-417c-9228-4611be3232c9","Type":"ContainerStarted","Data":"ec17d97822f424fab7e692b53bb40ea86fc1f48a72de876f8391f1ca54d548f7"}
Jul 16 22:21:07.436655 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:07.436567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dlt79" event={"ID":"971b2522-0c80-417c-9228-4611be3232c9","Type":"ContainerStarted","Data":"b4c4c8894e394da02c9d5cadb1e61618cce791ccfe13fc66bd353c6053dfb4bc"}
Jul 16 22:21:07.452789 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:07.452740    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dlt79" podStartSLOduration=1.528443428 podStartE2EDuration="4.452727159s" totalImagesPullingTime="2.924283731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:07.450913293 +0000 UTC m=+229.519236348" watchObservedRunningTime="2026-07-16 22:21:07.452727159 +0000 UTC m=+229.521050210"
Jul 16 22:21:08.139899 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.139865    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"]
Jul 16 22:21:08.142039 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.142025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.144190 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.144164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 22:21:08.144350 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.144190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 22:21:08.144815 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.144797    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-274xs\""
Jul 16 22:21:08.151510 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.151489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"]
Jul 16 22:21:08.277373 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.277339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d31da4d-b2c2-43c1-acf8-94c716a5a308-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.277552 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.277382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vl6\" (UniqueName: \"kubernetes.io/projected/2d31da4d-b2c2-43c1-acf8-94c716a5a308-kube-api-access-d9vl6\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.377904 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.377866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d31da4d-b2c2-43c1-acf8-94c716a5a308-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.378069 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.377914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vl6\" (UniqueName: \"kubernetes.io/projected/2d31da4d-b2c2-43c1-acf8-94c716a5a308-kube-api-access-d9vl6\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.378255 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.378233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d31da4d-b2c2-43c1-acf8-94c716a5a308-tmp\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.386063 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.386022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vl6\" (UniqueName: \"kubernetes.io/projected/2d31da4d-b2c2-43c1-acf8-94c716a5a308-kube-api-access-d9vl6\") pod \"openshift-lws-operator-fd8ccff4c-z7x6d\" (UID: \"2d31da4d-b2c2-43c1-acf8-94c716a5a308\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.450678 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.450602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"
Jul 16 22:21:08.576448 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:08.576424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d"]
Jul 16 22:21:08.579115 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:21:08.579059    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d31da4d_b2c2_43c1_acf8_94c716a5a308.slice/crio-0cfdd5bdd8be0e8a9abfcebab75b4c341f1824b9f774ccdb6130bc6a39752ddc WatchSource:0}: Error finding container 0cfdd5bdd8be0e8a9abfcebab75b4c341f1824b9f774ccdb6130bc6a39752ddc: Status 404 returned error can't find the container with id 0cfdd5bdd8be0e8a9abfcebab75b4c341f1824b9f774ccdb6130bc6a39752ddc
Jul 16 22:21:09.445404 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:09.445364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d" event={"ID":"2d31da4d-b2c2-43c1-acf8-94c716a5a308","Type":"ContainerStarted","Data":"0cfdd5bdd8be0e8a9abfcebab75b4c341f1824b9f774ccdb6130bc6a39752ddc"}
Jul 16 22:21:11.454202 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:11.454165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d" event={"ID":"2d31da4d-b2c2-43c1-acf8-94c716a5a308","Type":"ContainerStarted","Data":"784e41afe50122013cf9bd0e1ee0f0a2a636e579024a0906c99a674478649505"}
Jul 16 22:21:11.471523 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:11.471471    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-z7x6d" podStartSLOduration=0.719507103 podStartE2EDuration="3.471456431s" totalImagesPullingTime="2.751949328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:11.470671417 +0000 UTC m=+233.538994474" watchObservedRunningTime="2026-07-16 22:21:11.471456431 +0000 UTC m=+233.539779485"
Jul 16 22:21:29.337107 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.337075    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"]
Jul 16 22:21:29.343607 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.343577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.346489 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.346462    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 22:21:29.346489 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.346481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-592w7\""
Jul 16 22:21:29.346722 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.346508    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 22:21:29.346722 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.346535    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 22:21:29.346722 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.346476    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 22:21:29.360157 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.360127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"]
Jul 16 22:21:29.369885 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.369860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-tmp\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.370015 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.369895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.370015 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.369928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qtn\" (UniqueName: \"kubernetes.io/projected/4927ca4e-c54d-47a0-9c70-30d1be286cfc-kube-api-access-k5qtn\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.370015 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.369958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-manifests\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.370119 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.370018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.470840 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.470793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-tmp\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.471042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.470850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.471042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.470879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qtn\" (UniqueName: \"kubernetes.io/projected/4927ca4e-c54d-47a0-9c70-30d1be286cfc-kube-api-access-k5qtn\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.471042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.470905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-manifests\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.471042 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.470960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.471353 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.471329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-manifests\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.473208 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.473174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4927ca4e-c54d-47a0-9c70-30d1be286cfc-tmp\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.473337 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.473315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.473512 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.473491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4927ca4e-c54d-47a0-9c70-30d1be286cfc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.487650 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.487624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qtn\" (UniqueName: \"kubernetes.io/projected/4927ca4e-c54d-47a0-9c70-30d1be286cfc-kube-api-access-k5qtn\") pod \"opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj\" (UID: \"4927ca4e-c54d-47a0-9c70-30d1be286cfc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.661155 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.661054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:29.818881 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:21:29.818842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4927ca4e_c54d_47a0_9c70_30d1be286cfc.slice/crio-9f02514024bcac46206ac23fe6489d4c01fc52df1478872fd900e0ddc07f41f9 WatchSource:0}: Error finding container 9f02514024bcac46206ac23fe6489d4c01fc52df1478872fd900e0ddc07f41f9: Status 404 returned error can't find the container with id 9f02514024bcac46206ac23fe6489d4c01fc52df1478872fd900e0ddc07f41f9
Jul 16 22:21:29.829156 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:29.829129    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"]
Jul 16 22:21:30.526726 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:30.526670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj" event={"ID":"4927ca4e-c54d-47a0-9c70-30d1be286cfc","Type":"ContainerStarted","Data":"9f02514024bcac46206ac23fe6489d4c01fc52df1478872fd900e0ddc07f41f9"}
Jul 16 22:21:33.542681 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:33.542591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj" event={"ID":"4927ca4e-c54d-47a0-9c70-30d1be286cfc","Type":"ContainerStarted","Data":"331e8ad849863cd54409fc2f8ebc22c51f8981ef48cf9356f313d26dd54f2609"}
Jul 16 22:21:34.548446 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:34.548408    2579 generic.go:356] "Generic (PLEG): container finished" podID="4927ca4e-c54d-47a0-9c70-30d1be286cfc" containerID="331e8ad849863cd54409fc2f8ebc22c51f8981ef48cf9356f313d26dd54f2609" exitCode=0
Jul 16 22:21:34.548877 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:34.548468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj" event={"ID":"4927ca4e-c54d-47a0-9c70-30d1be286cfc","Type":"ContainerDied","Data":"331e8ad849863cd54409fc2f8ebc22c51f8981ef48cf9356f313d26dd54f2609"}
Jul 16 22:21:35.554174 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:35.554134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj" event={"ID":"4927ca4e-c54d-47a0-9c70-30d1be286cfc","Type":"ContainerStarted","Data":"5bc337ccf8468be6b91bb13ad37dc2d7948abe9fe3f5c87c7e5f12e5c638c2d7"}
Jul 16 22:21:35.554567 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:35.554240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:35.579836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:35.579783    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj" podStartSLOduration=1.741093389 podStartE2EDuration="6.579767659s" totalImagesPullingTime="3.675637424s" totalInitContainerRuntime="1.163036846s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:35.577838927 +0000 UTC m=+257.646161972" watchObservedRunningTime="2026-07-16 22:21:35.579767659 +0000 UTC m=+257.648090772"
Jul 16 22:21:46.560696 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.560610    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj"
Jul 16 22:21:46.952153 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.952059    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"]
Jul 16 22:21:46.955444 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.955422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:46.958162 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.958141    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 22:21:46.958631 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.958612    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 22:21:46.958943 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.958927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 22:21:46.959002 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.958927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xzcpf\""
Jul 16 22:21:46.966341 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:46.966311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"]
Jul 16 22:21:47.027191 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.027144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9257138b-ca56-4040-8c75-b8f4d8b9187b-manager-config\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.027385 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.027198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.027385 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.027279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsc54\" (UniqueName: \"kubernetes.io/projected/9257138b-ca56-4040-8c75-b8f4d8b9187b-kube-api-access-vsc54\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.027385 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.027310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-metrics-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.128497 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.128453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9257138b-ca56-4040-8c75-b8f4d8b9187b-manager-config\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.128497 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.128496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.128796 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.128630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsc54\" (UniqueName: \"kubernetes.io/projected/9257138b-ca56-4040-8c75-b8f4d8b9187b-kube-api-access-vsc54\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.128796 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.128674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-metrics-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.129227 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.129206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9257138b-ca56-4040-8c75-b8f4d8b9187b-manager-config\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.131076 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.131052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.131180 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.131086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9257138b-ca56-4040-8c75-b8f4d8b9187b-metrics-cert\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.138266 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.138241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsc54\" (UniqueName: \"kubernetes.io/projected/9257138b-ca56-4040-8c75-b8f4d8b9187b-kube-api-access-vsc54\") pod \"lws-controller-manager-67c6655946-mcr9w\" (UID: \"9257138b-ca56-4040-8c75-b8f4d8b9187b\") " pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.265434 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.265318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:47.400022 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.399987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"]
Jul 16 22:21:47.402700 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:21:47.402665    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9257138b_ca56_4040_8c75_b8f4d8b9187b.slice/crio-5df4d90c540ab786d81ebf95e6c7c4aae5ed9d6c3c569f6fd45cf6bca46df134 WatchSource:0}: Error finding container 5df4d90c540ab786d81ebf95e6c7c4aae5ed9d6c3c569f6fd45cf6bca46df134: Status 404 returned error can't find the container with id 5df4d90c540ab786d81ebf95e6c7c4aae5ed9d6c3c569f6fd45cf6bca46df134
Jul 16 22:21:47.604621 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:47.604590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w" event={"ID":"9257138b-ca56-4040-8c75-b8f4d8b9187b","Type":"ContainerStarted","Data":"5df4d90c540ab786d81ebf95e6c7c4aae5ed9d6c3c569f6fd45cf6bca46df134"}
Jul 16 22:21:49.154691 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.154653    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"]
Jul 16 22:21:49.158537 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.158516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.164477 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.164454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 22:21:49.165056 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.165032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cnqmk\""
Jul 16 22:21:49.165320 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.165300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 22:21:49.165320 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.165312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 22:21:49.167407 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.167387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 22:21:49.174977 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.174950    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"]
Jul 16 22:21:49.248724 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.248686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvws7\" (UniqueName: \"kubernetes.io/projected/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-kube-api-access-dvws7\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.248927 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.248763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tmp\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.248927 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.248903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tls-certs\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.349590 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.349552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvws7\" (UniqueName: \"kubernetes.io/projected/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-kube-api-access-dvws7\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.349782 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.349613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tmp\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.349782 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.349677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tls-certs\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.351978 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.351945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tmp\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.352187 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.352172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-tls-certs\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.375522 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.375497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvws7\" (UniqueName: \"kubernetes.io/projected/1d9a09f9-a705-4ca7-b9ab-002e598b7db3-kube-api-access-dvws7\") pod \"kube-auth-proxy-7b969c6b8c-kdw76\" (UID: \"1d9a09f9-a705-4ca7-b9ab-002e598b7db3\") " pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.468796 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.468759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"
Jul 16 22:21:49.601405 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.601381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76"]
Jul 16 22:21:49.603629 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:21:49.603602    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9a09f9_a705_4ca7_b9ab_002e598b7db3.slice/crio-c141ff574ccb704cceb90ebf612386b3138fcce397cd07198cea93d932479900 WatchSource:0}: Error finding container c141ff574ccb704cceb90ebf612386b3138fcce397cd07198cea93d932479900: Status 404 returned error can't find the container with id c141ff574ccb704cceb90ebf612386b3138fcce397cd07198cea93d932479900
Jul 16 22:21:49.616752 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.616699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w" event={"ID":"9257138b-ca56-4040-8c75-b8f4d8b9187b","Type":"ContainerStarted","Data":"d502868e06ba766b51115eb67b96ccdecf25825f39685218be13092f18c6ee51"}
Jul 16 22:21:49.616864 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.616779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:21:49.617981 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.617958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76" event={"ID":"1d9a09f9-a705-4ca7-b9ab-002e598b7db3","Type":"ContainerStarted","Data":"c141ff574ccb704cceb90ebf612386b3138fcce397cd07198cea93d932479900"}
Jul 16 22:21:49.639380 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:49.639325    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w" podStartSLOduration=1.9990979690000001 podStartE2EDuration="3.639308077s" totalImagesPullingTime="1.640210108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:49.638574521 +0000 UTC m=+271.706897576" watchObservedRunningTime="2026-07-16 22:21:49.639308077 +0000 UTC m=+271.707631131"
Jul 16 22:21:52.633819 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:52.633784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76" event={"ID":"1d9a09f9-a705-4ca7-b9ab-002e598b7db3","Type":"ContainerStarted","Data":"ff90e1d5b668f3504de7d2fea32adfba025a48d28ec3e392a63adec0f6116a9b"}
Jul 16 22:21:52.650993 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:21:52.650942    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b969c6b8c-kdw76" podStartSLOduration=0.713533843 podStartE2EDuration="3.650925521s" totalImagesPullingTime="2.937391678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:52.649345168 +0000 UTC m=+274.717668223" watchObservedRunningTime="2026-07-16 22:21:52.650925521 +0000 UTC m=+274.719248574"
Jul 16 22:22:00.624025 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:00.623988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-67c6655946-mcr9w"
Jul 16 22:22:01.285022 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.284987    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"]
Jul 16 22:22:01.288412 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.288395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.296849 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.296825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 22:22:01.296970 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.296828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pf85h\""
Jul 16 22:22:01.305886 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.305857    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"]
Jul 16 22:22:01.463607 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.463569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmbk\" (UniqueName: \"kubernetes.io/projected/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-kube-api-access-fkmbk\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.463607 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.463615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.564582 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.564554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.564756 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.564640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmbk\" (UniqueName: \"kubernetes.io/projected/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-kube-api-access-fkmbk\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.566993 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.566970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.574207 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.574181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmbk\" (UniqueName: \"kubernetes.io/projected/1a53e0ad-e5b4-400c-80e7-9b0b416a23c1-kube-api-access-fkmbk\") pod \"llmisvc-controller-manager-777c6bd7d5-28m6q\" (UID: \"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.597926 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.597905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:01.755386 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:01.755351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"]
Jul 16 22:22:01.759433 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:22:01.759405    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a53e0ad_e5b4_400c_80e7_9b0b416a23c1.slice/crio-e617d10444e46237a32032e2ed7ccb19881414bce86ab0a9cada45fddbc353f6 WatchSource:0}: Error finding container e617d10444e46237a32032e2ed7ccb19881414bce86ab0a9cada45fddbc353f6: Status 404 returned error can't find the container with id e617d10444e46237a32032e2ed7ccb19881414bce86ab0a9cada45fddbc353f6
Jul 16 22:22:02.681062 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:02.681027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q" event={"ID":"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1","Type":"ContainerStarted","Data":"e617d10444e46237a32032e2ed7ccb19881414bce86ab0a9cada45fddbc353f6"}
Jul 16 22:22:05.697554 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:05.697517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q" event={"ID":"1a53e0ad-e5b4-400c-80e7-9b0b416a23c1","Type":"ContainerStarted","Data":"04c96d75549eb4785f09a5379f5a9dae23d73924575e22da56dc58c2017d4f28"}
Jul 16 22:22:05.697971 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:05.697660    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:22:05.715600 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:05.715546    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q" podStartSLOduration=1.298976477 podStartE2EDuration="4.715530613s" totalImagesPullingTime="3.416554136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:05.715380176 +0000 UTC m=+287.783703230" watchObservedRunningTime="2026-07-16 22:22:05.715530613 +0000 UTC m=+287.783853666"
Jul 16 22:22:18.309031 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:18.309006    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 22:22:36.705001 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:22:36.704972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-28m6q"
Jul 16 22:23:50.215187 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.215151    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"]
Jul 16 22:23:50.218501 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.218483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.222321 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.222300    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:23:50.222449 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.222394    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:23:50.222503 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.222458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-shwv8\""
Jul 16 22:23:50.239246 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.239213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"]
Jul 16 22:23:50.391120 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.391084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.391120 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.391132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnh55\" (UniqueName: \"kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.491973 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.491873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.491973 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.491912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnh55\" (UniqueName: \"kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.492334 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.492312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.499675 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.499646    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"]
Jul 16 22:23:50.499929 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:23:50.499911    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jnh55], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2"
Jul 16 22:23:50.515503 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.515447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnh55\" (UniqueName: \"kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55\") pod \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:50.520543 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:50.520515    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"]
Jul 16 22:23:51.130094 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.130056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:51.132102 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:23:51.132065    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:51.137331 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.137307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:51.142257 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:23:51.142231    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:51.300145 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.300112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume\") pod \"e2a4c498-6ced-4188-8446-72209bf03ef2\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") "
Jul 16 22:23:51.300625 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.300226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnh55\" (UniqueName: \"kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55\") pod \"e2a4c498-6ced-4188-8446-72209bf03ef2\" (UID: \"e2a4c498-6ced-4188-8446-72209bf03ef2\") "
Jul 16 22:23:51.300625 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.300355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e2a4c498-6ced-4188-8446-72209bf03ef2" (UID: "e2a4c498-6ced-4188-8446-72209bf03ef2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:23:51.300625 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.300419    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e2a4c498-6ced-4188-8446-72209bf03ef2-extensions-socket-volume\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:23:51.302332 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.302306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55" (OuterVolumeSpecName: "kube-api-access-jnh55") pod "e2a4c498-6ced-4188-8446-72209bf03ef2" (UID: "e2a4c498-6ced-4188-8446-72209bf03ef2"). InnerVolumeSpecName "kube-api-access-jnh55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:23:51.401425 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:51.401333    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnh55\" (UniqueName: \"kubernetes.io/projected/e2a4c498-6ced-4188-8446-72209bf03ef2-kube-api-access-jnh55\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:23:52.134545 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:52.134512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:52.136938 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:23:52.136907    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:52.142001 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:23:52.141975    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d9vl6\" is forbidden: User \"system:node:ip-10-0-141-9.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-9.ec2.internal' and this object" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d9vl6"
Jul 16 22:23:52.462126 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:23:52.462048    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a4c498-6ced-4188-8446-72209bf03ef2" path="/var/lib/kubelet/pods/e2a4c498-6ced-4188-8446-72209bf03ef2/volumes"
Jul 16 22:24:38.217951 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.217913    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:38.221252 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.221233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.223888 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.223865    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:24:38.224022 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.223948    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 22:24:38.224392 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.224378    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r9mf7\""
Jul 16 22:24:38.224440 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.224378    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:24:38.229331 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.229312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:38.314176 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.314144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:38.409536 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.409494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.409753 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.409558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.510424 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.510330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.510424 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.510376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.511076 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.511057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.519258 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.519232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl\") pod \"limitador-limitador-64c94fd64f-mxc5r\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.532199 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.532179    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:38.684631 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.684602    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:38.686382 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:24:38.686341    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725ab6d9_781e_4494_af0a_9d1be0886cd8.slice/crio-8f6b44e20e060d008c7f9db3600824c390f9a2d12c3180f4f53fdfd29fac501a WatchSource:0}: Error finding container 8f6b44e20e060d008c7f9db3600824c390f9a2d12c3180f4f53fdfd29fac501a: Status 404 returned error can't find the container with id 8f6b44e20e060d008c7f9db3600824c390f9a2d12c3180f4f53fdfd29fac501a
Jul 16 22:24:38.688185 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:38.688167    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:24:39.049647 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.049612    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:39.053870 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.053851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:39.056828 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.056794    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvr2t\""
Jul 16 22:24:39.058826 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.058810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:39.217958 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.217925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jd7\" (UniqueName: \"kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7\") pod \"authorino-58cd955675-6nsr6\" (UID: \"f8f3896c-4999-4c91-a1c1-34402383105c\") " pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:39.319091 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.319057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jd7\" (UniqueName: \"kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7\") pod \"authorino-58cd955675-6nsr6\" (UID: \"f8f3896c-4999-4c91-a1c1-34402383105c\") " pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:39.331422 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.331393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jd7\" (UniqueName: \"kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7\") pod \"authorino-58cd955675-6nsr6\" (UID: \"f8f3896c-4999-4c91-a1c1-34402383105c\") " pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:39.345268 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.345221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" event={"ID":"725ab6d9-781e-4494-af0a-9d1be0886cd8","Type":"ContainerStarted","Data":"8f6b44e20e060d008c7f9db3600824c390f9a2d12c3180f4f53fdfd29fac501a"}
Jul 16 22:24:39.364126 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.364096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:39.592617 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:39.591918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:39.604845 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:24:39.604798    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f3896c_4999_4c91_a1c1_34402383105c.slice/crio-9f1135fa4a06b3bcdc5de88e3cf4d7db9ad2b2b4ae202415b3d34901f53126b0 WatchSource:0}: Error finding container 9f1135fa4a06b3bcdc5de88e3cf4d7db9ad2b2b4ae202415b3d34901f53126b0: Status 404 returned error can't find the container with id 9f1135fa4a06b3bcdc5de88e3cf4d7db9ad2b2b4ae202415b3d34901f53126b0
Jul 16 22:24:40.352492 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:40.352451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6nsr6" event={"ID":"f8f3896c-4999-4c91-a1c1-34402383105c","Type":"ContainerStarted","Data":"9f1135fa4a06b3bcdc5de88e3cf4d7db9ad2b2b4ae202415b3d34901f53126b0"}
Jul 16 22:24:42.365563 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:42.365524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" event={"ID":"725ab6d9-781e-4494-af0a-9d1be0886cd8","Type":"ContainerStarted","Data":"00c04db5221b721638d4ea74f69a295bdad4feb0c787b4aa6edf8a2311441288"}
Jul 16 22:24:42.366031 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:42.365618    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:42.383793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:42.383735    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" podStartSLOduration=1.594377521 podStartE2EDuration="4.383700029s" totalImagesPullingTime="2.789322508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:42.382206922 +0000 UTC m=+444.450529982" watchObservedRunningTime="2026-07-16 22:24:42.383700029 +0000 UTC m=+444.452023085"
Jul 16 22:24:44.191810 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:44.191777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:44.376175 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:44.376143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6nsr6" event={"ID":"f8f3896c-4999-4c91-a1c1-34402383105c","Type":"ContainerStarted","Data":"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"}
Jul 16 22:24:44.391675 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:44.391621    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-6nsr6" podStartSLOduration=1.255971431 podStartE2EDuration="5.391601437s" totalImagesPullingTime="4.135630006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:44.390213547 +0000 UTC m=+446.458536601" watchObservedRunningTime="2026-07-16 22:24:44.391601437 +0000 UTC m=+446.459924494"
Jul 16 22:24:45.380188 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:45.380133    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-6nsr6" podUID="f8f3896c-4999-4c91-a1c1-34402383105c" containerName="authorino" containerID="cri-o://f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6" gracePeriod=30
Jul 16 22:24:45.625741 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:45.625697    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:45.682228 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:45.682146    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72jd7\" (UniqueName: \"kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7\") pod \"f8f3896c-4999-4c91-a1c1-34402383105c\" (UID: \"f8f3896c-4999-4c91-a1c1-34402383105c\") "
Jul 16 22:24:45.684143 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:45.684110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7" (OuterVolumeSpecName: "kube-api-access-72jd7") pod "f8f3896c-4999-4c91-a1c1-34402383105c" (UID: "f8f3896c-4999-4c91-a1c1-34402383105c"). InnerVolumeSpecName "kube-api-access-72jd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:24:45.782869 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:45.782831    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jd7\" (UniqueName: \"kubernetes.io/projected/f8f3896c-4999-4c91-a1c1-34402383105c-kube-api-access-72jd7\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:24:46.385537 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.385498    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8f3896c-4999-4c91-a1c1-34402383105c" containerID="f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6" exitCode=0
Jul 16 22:24:46.386008 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.385544    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6nsr6"
Jul 16 22:24:46.386008 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.385587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6nsr6" event={"ID":"f8f3896c-4999-4c91-a1c1-34402383105c","Type":"ContainerDied","Data":"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"}
Jul 16 22:24:46.386008 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.385631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6nsr6" event={"ID":"f8f3896c-4999-4c91-a1c1-34402383105c","Type":"ContainerDied","Data":"9f1135fa4a06b3bcdc5de88e3cf4d7db9ad2b2b4ae202415b3d34901f53126b0"}
Jul 16 22:24:46.386008 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.385648    2579 scope.go:117] "RemoveContainer" containerID="f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"
Jul 16 22:24:46.394648 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.394629    2579 scope.go:117] "RemoveContainer" containerID="f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"
Jul 16 22:24:46.394944 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:24:46.394923    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6\": container with ID starting with f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6 not found: ID does not exist" containerID="f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"
Jul 16 22:24:46.394997 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.394954    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6"} err="failed to get container status \"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6\": rpc error: code = NotFound desc = could not find container \"f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6\": container with ID starting with f4a3f0363e03b2a8d2ef9d7945c7c36803d36564732776efe7347dba76e4f2d6 not found: ID does not exist"
Jul 16 22:24:46.409210 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.409184    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:46.413401 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.413380    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-6nsr6"]
Jul 16 22:24:46.467030 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:46.467002    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f3896c-4999-4c91-a1c1-34402383105c" path="/var/lib/kubelet/pods/f8f3896c-4999-4c91-a1c1-34402383105c/volumes"
Jul 16 22:24:53.372176 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:53.372147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:53.884260 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:53.884216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:53.884526 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:53.884487    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" podUID="725ab6d9-781e-4494-af0a-9d1be0886cd8" containerName="limitador" containerID="cri-o://00c04db5221b721638d4ea74f69a295bdad4feb0c787b4aa6edf8a2311441288" gracePeriod=30
Jul 16 22:24:54.421479 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.421443    2579 generic.go:356] "Generic (PLEG): container finished" podID="725ab6d9-781e-4494-af0a-9d1be0886cd8" containerID="00c04db5221b721638d4ea74f69a295bdad4feb0c787b4aa6edf8a2311441288" exitCode=0
Jul 16 22:24:54.421871 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.421518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" event={"ID":"725ab6d9-781e-4494-af0a-9d1be0886cd8","Type":"ContainerDied","Data":"00c04db5221b721638d4ea74f69a295bdad4feb0c787b4aa6edf8a2311441288"}
Jul 16 22:24:54.843289 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.843266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:54.962211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.962125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl\") pod \"725ab6d9-781e-4494-af0a-9d1be0886cd8\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") "
Jul 16 22:24:54.962211 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.962169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file\") pod \"725ab6d9-781e-4494-af0a-9d1be0886cd8\" (UID: \"725ab6d9-781e-4494-af0a-9d1be0886cd8\") "
Jul 16 22:24:54.962562 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.962540    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file" (OuterVolumeSpecName: "config-file") pod "725ab6d9-781e-4494-af0a-9d1be0886cd8" (UID: "725ab6d9-781e-4494-af0a-9d1be0886cd8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:54.964220 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:54.964196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl" (OuterVolumeSpecName: "kube-api-access-dsrvl") pod "725ab6d9-781e-4494-af0a-9d1be0886cd8" (UID: "725ab6d9-781e-4494-af0a-9d1be0886cd8"). InnerVolumeSpecName "kube-api-access-dsrvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:24:55.062764 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.062698    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsrvl\" (UniqueName: \"kubernetes.io/projected/725ab6d9-781e-4494-af0a-9d1be0886cd8-kube-api-access-dsrvl\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:24:55.062764 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.062756    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/725ab6d9-781e-4494-af0a-9d1be0886cd8-config-file\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:24:55.427153 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.427115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r" event={"ID":"725ab6d9-781e-4494-af0a-9d1be0886cd8","Type":"ContainerDied","Data":"8f6b44e20e060d008c7f9db3600824c390f9a2d12c3180f4f53fdfd29fac501a"}
Jul 16 22:24:55.427153 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.427143    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"
Jul 16 22:24:55.427767 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.427178    2579 scope.go:117] "RemoveContainer" containerID="00c04db5221b721638d4ea74f69a295bdad4feb0c787b4aa6edf8a2311441288"
Jul 16 22:24:55.449308 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.449286    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:55.456112 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:55.456090    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mxc5r"]
Jul 16 22:24:56.461753 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:56.461720    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725ab6d9-781e-4494-af0a-9d1be0886cd8" path="/var/lib/kubelet/pods/725ab6d9-781e-4494-af0a-9d1be0886cd8/volumes"
Jul 16 22:24:57.726888 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.726852    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"]
Jul 16 22:24:57.727251 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f3896c-4999-4c91-a1c1-34402383105c" containerName="authorino"
Jul 16 22:24:57.727251 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727225    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f3896c-4999-4c91-a1c1-34402383105c" containerName="authorino"
Jul 16 22:24:57.727251 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727247    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725ab6d9-781e-4494-af0a-9d1be0886cd8" containerName="limitador"
Jul 16 22:24:57.727349 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727255    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="725ab6d9-781e-4494-af0a-9d1be0886cd8" containerName="limitador"
Jul 16 22:24:57.727349 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727317    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f3896c-4999-4c91-a1c1-34402383105c" containerName="authorino"
Jul 16 22:24:57.727349 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.727330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="725ab6d9-781e-4494-af0a-9d1be0886cd8" containerName="limitador"
Jul 16 22:24:57.731553 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.731532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.734371 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.734351    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 16 22:24:57.734516 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.734411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 16 22:24:57.734516 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.734454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:57.734872 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.734858    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-shzxh\""
Jul 16 22:24:57.739307 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.739286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"]
Jul 16 22:24:57.787020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.786982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngz55\" (UniqueName: \"kubernetes.io/projected/b50e2c63-af4d-46c4-96d9-61505413be53-kube-api-access-ngz55\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.787020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.787022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b50e2c63-af4d-46c4-96d9-61505413be53-data\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.888117 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.888082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngz55\" (UniqueName: \"kubernetes.io/projected/b50e2c63-af4d-46c4-96d9-61505413be53-kube-api-access-ngz55\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.888276 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.888127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b50e2c63-af4d-46c4-96d9-61505413be53-data\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.888555 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.888534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/b50e2c63-af4d-46c4-96d9-61505413be53-data\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:57.896620 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:57.896599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngz55\" (UniqueName: \"kubernetes.io/projected/b50e2c63-af4d-46c4-96d9-61505413be53-kube-api-access-ngz55\") pod \"postgres-69fb4594cf-s6k7g\" (UID: \"b50e2c63-af4d-46c4-96d9-61505413be53\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:58.042454 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:58.042369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:24:58.175244 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:58.175212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"]
Jul 16 22:24:58.177958 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:24:58.177929    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50e2c63_af4d_46c4_96d9_61505413be53.slice/crio-0de07e64015239d6cb44ceafa7ba1e2985e110abe42ba5339d1af068e768ee6a WatchSource:0}: Error finding container 0de07e64015239d6cb44ceafa7ba1e2985e110abe42ba5339d1af068e768ee6a: Status 404 returned error can't find the container with id 0de07e64015239d6cb44ceafa7ba1e2985e110abe42ba5339d1af068e768ee6a
Jul 16 22:24:58.441467 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:24:58.441432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g" event={"ID":"b50e2c63-af4d-46c4-96d9-61505413be53","Type":"ContainerStarted","Data":"0de07e64015239d6cb44ceafa7ba1e2985e110abe42ba5339d1af068e768ee6a"}
Jul 16 22:25:03.469202 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:03.469108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g" event={"ID":"b50e2c63-af4d-46c4-96d9-61505413be53","Type":"ContainerStarted","Data":"928a4f0c5337b285eb4e525586e44ce4e800ce4bbea50434e0f94c7dcc418ac3"}
Jul 16 22:25:03.469202 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:03.469169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:25:03.489322 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:03.489272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g" podStartSLOduration=1.564135173 podStartE2EDuration="6.489255725s" totalImagesPullingTime="4.925120552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:03.487447964 +0000 UTC m=+465.555771021" watchObservedRunningTime="2026-07-16 22:25:03.489255725 +0000 UTC m=+465.557578779"
Jul 16 22:25:09.502971 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:09.502943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s6k7g"
Jul 16 22:25:10.044492 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.044452    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7pc4"]
Jul 16 22:25:10.047460 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.047436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.049798 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.049775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvr2t\""
Jul 16 22:25:10.049940 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.049922    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 22:25:10.050456 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.050441    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 22:25:10.054839 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.054815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7pc4"]
Jul 16 22:25:10.204364 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.204322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bxf\" (UniqueName: \"kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf\") pod \"authorino-5d4b67c446-d7pc4\" (UID: \"6c93222a-bc74-4a2a-9af3-d1ddc54c03b3\") " pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.267360 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.267318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7pc4"]
Jul 16 22:25:10.267631 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:25:10.267608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-24bxf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-d7pc4" podUID="6c93222a-bc74-4a2a-9af3-d1ddc54c03b3"
Jul 16 22:25:10.305195 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.305108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bxf\" (UniqueName: \"kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf\") pod \"authorino-5d4b67c446-d7pc4\" (UID: \"6c93222a-bc74-4a2a-9af3-d1ddc54c03b3\") " pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.316902 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.316855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bxf\" (UniqueName: \"kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf\") pod \"authorino-5d4b67c446-d7pc4\" (UID: \"6c93222a-bc74-4a2a-9af3-d1ddc54c03b3\") " pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.498227 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.498188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.505049 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.505026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:10.598409 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.598375    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:25:10.601327 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.601310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.603658 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.603640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 16 22:25:10.607791 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.607768    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24bxf\" (UniqueName: \"kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf\") pod \"6c93222a-bc74-4a2a-9af3-d1ddc54c03b3\" (UID: \"6c93222a-bc74-4a2a-9af3-d1ddc54c03b3\") "
Jul 16 22:25:10.610549 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.610519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf" (OuterVolumeSpecName: "kube-api-access-24bxf") pod "6c93222a-bc74-4a2a-9af3-d1ddc54c03b3" (UID: "6c93222a-bc74-4a2a-9af3-d1ddc54c03b3"). InnerVolumeSpecName "kube-api-access-24bxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:25:10.611469 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.611448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:25:10.708952 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.708907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.708952 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.708956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnxh\" (UniqueName: \"kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.709163 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.709095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bxf\" (UniqueName: \"kubernetes.io/projected/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3-kube-api-access-24bxf\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:25:10.809841 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.809795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.809841 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.809843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnxh\" (UniqueName: \"kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.812336 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.812305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.817789 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.817766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnxh\" (UniqueName: \"kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh\") pod \"authorino-79c66c87dc-kz7t2\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") " pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:10.910771 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:10.910644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:25:11.044317 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:11.044290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:25:11.046365 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:25:11.046337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod195b1361_63b8_4e04_a6eb_1287a1f89131.slice/crio-ff5f96b9d05e1245e806c84fbd7af46108f88d35e7a32dc697761b3b3e5271e3 WatchSource:0}: Error finding container ff5f96b9d05e1245e806c84fbd7af46108f88d35e7a32dc697761b3b3e5271e3: Status 404 returned error can't find the container with id ff5f96b9d05e1245e806c84fbd7af46108f88d35e7a32dc697761b3b3e5271e3
Jul 16 22:25:11.503341 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:11.503302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" event={"ID":"195b1361-63b8-4e04-a6eb-1287a1f89131","Type":"ContainerStarted","Data":"ff5f96b9d05e1245e806c84fbd7af46108f88d35e7a32dc697761b3b3e5271e3"}
Jul 16 22:25:11.503341 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:11.503315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-d7pc4"
Jul 16 22:25:11.535049 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:11.535016    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7pc4"]
Jul 16 22:25:11.537052 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:11.537028    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-d7pc4"]
Jul 16 22:25:12.463491 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:12.463461    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c93222a-bc74-4a2a-9af3-d1ddc54c03b3" path="/var/lib/kubelet/pods/6c93222a-bc74-4a2a-9af3-d1ddc54c03b3/volumes"
Jul 16 22:25:12.508598 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:12.508565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" event={"ID":"195b1361-63b8-4e04-a6eb-1287a1f89131","Type":"ContainerStarted","Data":"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"}
Jul 16 22:25:12.526297 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:12.526244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" podStartSLOduration=1.28988701 podStartE2EDuration="2.526226347s" totalImagesPullingTime="1.236339337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:12.524326291 +0000 UTC m=+474.592649346" watchObservedRunningTime="2026-07-16 22:25:12.526226347 +0000 UTC m=+474.594549401"
Jul 16 22:25:53.531281 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.531249    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-7jm7k"]
Jul 16 22:25:53.533521 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.533502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.535754 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.535732    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 16 22:25:53.535873 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.535825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-k7xts\""
Jul 16 22:25:53.543533 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.543508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-7jm7k"]
Jul 16 22:25:53.586331 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.586292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkxg\" (UniqueName: \"kubernetes.io/projected/765eb737-62ad-408f-bdc5-fa9dd064c524-kube-api-access-vdkxg\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.586533 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.586345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765eb737-62ad-408f-bdc5-fa9dd064c524-plugins-config-volume\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.687539 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.687506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkxg\" (UniqueName: \"kubernetes.io/projected/765eb737-62ad-408f-bdc5-fa9dd064c524-kube-api-access-vdkxg\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.687739 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.687566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765eb737-62ad-408f-bdc5-fa9dd064c524-plugins-config-volume\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.688301 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.688283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/765eb737-62ad-408f-bdc5-fa9dd064c524-plugins-config-volume\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.698036 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.698014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkxg\" (UniqueName: \"kubernetes.io/projected/765eb737-62ad-408f-bdc5-fa9dd064c524-kube-api-access-vdkxg\") pod \"payload-processing-5bf47cd988-7jm7k\" (UID: \"765eb737-62ad-408f-bdc5-fa9dd064c524\") " pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.844571 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.844533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:53.976185 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:53.976161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-7jm7k"]
Jul 16 22:25:53.978536 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:25:53.978507    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765eb737_62ad_408f_bdc5_fa9dd064c524.slice/crio-d71dac8b00283bc94bce9ea526836d0e0e42264d5f8cdaa0f7b132cfd9354aed WatchSource:0}: Error finding container d71dac8b00283bc94bce9ea526836d0e0e42264d5f8cdaa0f7b132cfd9354aed: Status 404 returned error can't find the container with id d71dac8b00283bc94bce9ea526836d0e0e42264d5f8cdaa0f7b132cfd9354aed
Jul 16 22:25:54.687438 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:54.687403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k" event={"ID":"765eb737-62ad-408f-bdc5-fa9dd064c524","Type":"ContainerStarted","Data":"d71dac8b00283bc94bce9ea526836d0e0e42264d5f8cdaa0f7b132cfd9354aed"}
Jul 16 22:25:56.699577 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:56.699539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k" event={"ID":"765eb737-62ad-408f-bdc5-fa9dd064c524","Type":"ContainerStarted","Data":"7d52153391fae2d3aabe1f451beb7b74873160186a603840567d280daa4db135"}
Jul 16 22:25:56.699995 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:56.699656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:25:56.719678 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:25:56.719620    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k" podStartSLOduration=1.843940288 podStartE2EDuration="3.719601634s" totalImagesPullingTime="1.875661346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:56.71693714 +0000 UTC m=+518.785260206" watchObservedRunningTime="2026-07-16 22:25:56.719601634 +0000 UTC m=+518.787924690"
Jul 16 22:26:07.706524 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:07.706490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-7jm7k"
Jul 16 22:26:10.061677 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.061595    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"]
Jul 16 22:26:10.065530 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.065504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.068281 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.068237    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 22:26:10.068977 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.068958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zttcm\""
Jul 16 22:26:10.069105 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.068991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 16 22:26:10.069105 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.069029    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 22:26:10.077249 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.077222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"]
Jul 16 22:26:10.233570 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.233836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.233836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.233836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.233836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.233836 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.233776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kth8n\" (UniqueName: \"kubernetes.io/projected/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-kube-api-access-kth8n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334532 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334532 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334532 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334532 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334883 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334883 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kth8n\" (UniqueName: \"kubernetes.io/projected/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-kube-api-access-kth8n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.334984 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.335034 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.334979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.335072 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.335057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.336971 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.336940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.337110 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.337093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.343167 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.343143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kth8n\" (UniqueName: \"kubernetes.io/projected/7332ab36-1b3b-4867-bb53-5b503d6fc5fa-kube-api-access-kth8n\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9\" (UID: \"7332ab36-1b3b-4867-bb53-5b503d6fc5fa\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.377266 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.377213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:10.515775 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.515750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"]
Jul 16 22:26:10.518492 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:26:10.518454    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7332ab36_1b3b_4867_bb53_5b503d6fc5fa.slice/crio-ea6f7f217bb88053f633e8f9cc99a2b0c33d43cb0910bc75781fe94be6bb3852 WatchSource:0}: Error finding container ea6f7f217bb88053f633e8f9cc99a2b0c33d43cb0910bc75781fe94be6bb3852: Status 404 returned error can't find the container with id ea6f7f217bb88053f633e8f9cc99a2b0c33d43cb0910bc75781fe94be6bb3852
Jul 16 22:26:10.763124 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:10.763035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9" event={"ID":"7332ab36-1b3b-4867-bb53-5b503d6fc5fa","Type":"ContainerStarted","Data":"ea6f7f217bb88053f633e8f9cc99a2b0c33d43cb0910bc75781fe94be6bb3852"}
Jul 16 22:26:11.769770 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:11.769735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9" event={"ID":"7332ab36-1b3b-4867-bb53-5b503d6fc5fa","Type":"ContainerStarted","Data":"b96abf7b22c3bd4d3e8050d87836a0cc3143e546f3e43cfeaeeffb6883867bba"}
Jul 16 22:26:11.787621 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:11.787565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9" podStartSLOduration=0.624972732 podStartE2EDuration="1.787543154s" totalImagesPullingTime="1.162570422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:11.787192119 +0000 UTC m=+533.855515174" watchObservedRunningTime="2026-07-16 22:26:11.787543154 +0000 UTC m=+533.855866209"
Jul 16 22:26:20.377803 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:20.377762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:20.378233 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:20.377859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:20.390497 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:20.390470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:20.822842 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:20.822817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9"
Jul 16 22:26:40.391445 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.391403    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"]
Jul 16 22:26:40.396983 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.396963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.399420 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.399398    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 16 22:26:40.406504 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.406484    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"]
Jul 16 22:26:40.503470 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.503654 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42de968-bdaa-4a01-bf64-c1147381736e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.503654 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.503654 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.503852 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwxh\" (UniqueName: \"kubernetes.io/projected/c42de968-bdaa-4a01-bf64-c1147381736e-kube-api-access-8wwxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.503852 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.503699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604421 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42de968-bdaa-4a01-bf64-c1147381736e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604600 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604600 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604679 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwxh\" (UniqueName: \"kubernetes.io/projected/c42de968-bdaa-4a01-bf64-c1147381736e-kube-api-access-8wwxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604748 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604748 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.604847 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.604814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.605075 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.605048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.605152 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.605072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.606807 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.606789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42de968-bdaa-4a01-bf64-c1147381736e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.607114 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.607077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42de968-bdaa-4a01-bf64-c1147381736e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.612755 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.612700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwxh\" (UniqueName: \"kubernetes.io/projected/c42de968-bdaa-4a01-bf64-c1147381736e-kube-api-access-8wwxh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-fmkr2\" (UID: \"c42de968-bdaa-4a01-bf64-c1147381736e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.709110 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.709022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:40.858282 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.858253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"]
Jul 16 22:26:40.861557 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:26:40.861523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42de968_bdaa_4a01_bf64_c1147381736e.slice/crio-927f0b121780018c401c446f8c0f00970f0e6cb904de8e601a0a3a2ec5a23df3 WatchSource:0}: Error finding container 927f0b121780018c401c446f8c0f00970f0e6cb904de8e601a0a3a2ec5a23df3: Status 404 returned error can't find the container with id 927f0b121780018c401c446f8c0f00970f0e6cb904de8e601a0a3a2ec5a23df3
Jul 16 22:26:40.908158 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:40.908123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2" event={"ID":"c42de968-bdaa-4a01-bf64-c1147381736e","Type":"ContainerStarted","Data":"927f0b121780018c401c446f8c0f00970f0e6cb904de8e601a0a3a2ec5a23df3"}
Jul 16 22:26:41.914117 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:41.914082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2" event={"ID":"c42de968-bdaa-4a01-bf64-c1147381736e","Type":"ContainerStarted","Data":"6632ab5a191b8f1de16abd236238a79a92687436df6aeeb022ce46c3b98c574b"}
Jul 16 22:26:41.932545 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:41.932493    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2" podStartSLOduration=1.757626756 podStartE2EDuration="1.932478741s" totalImagesPullingTime="174.851985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:41.930469293 +0000 UTC m=+563.998792348" watchObservedRunningTime="2026-07-16 22:26:41.932478741 +0000 UTC m=+564.000801795"
Jul 16 22:26:50.709985 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:50.709940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:50.710536 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:50.710003    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:50.722395 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:50.722372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:26:50.973568 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:26:50.973497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-fmkr2"
Jul 16 22:27:19.577545 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.577508    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"]
Jul 16 22:27:19.581341 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.581323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.583530 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.583514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 16 22:27:19.590985 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.590960    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"]
Jul 16 22:27:19.646770 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flx56\" (UniqueName: \"kubernetes.io/projected/5906fa26-ee75-4260-aecf-32245c75dd17-kube-api-access-flx56\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.646770 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.646952 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.646952 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.646952 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5906fa26-ee75-4260-aecf-32245c75dd17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.647051 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.646972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748195 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flx56\" (UniqueName: \"kubernetes.io/projected/5906fa26-ee75-4260-aecf-32245c75dd17-kube-api-access-flx56\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748363 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748363 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748363 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748363 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5906fa26-ee75-4260-aecf-32245c75dd17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748521 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748724 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748793 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.748843 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.748812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.750474 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.750451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5906fa26-ee75-4260-aecf-32245c75dd17-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.750863 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.750841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5906fa26-ee75-4260-aecf-32245c75dd17-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.756020 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.756001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flx56\" (UniqueName: \"kubernetes.io/projected/5906fa26-ee75-4260-aecf-32245c75dd17-kube-api-access-flx56\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx\" (UID: \"5906fa26-ee75-4260-aecf-32245c75dd17\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:19.890360 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:19.890283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:20.021885 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:20.021787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"]
Jul 16 22:27:20.024004 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:27:20.023969    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5906fa26_ee75_4260_aecf_32245c75dd17.slice/crio-868e1d678c4889b94626e086f9dcbd6745639745583b02c4f0caa2225880e32b WatchSource:0}: Error finding container 868e1d678c4889b94626e086f9dcbd6745639745583b02c4f0caa2225880e32b: Status 404 returned error can't find the container with id 868e1d678c4889b94626e086f9dcbd6745639745583b02c4f0caa2225880e32b
Jul 16 22:27:20.088491 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:20.088457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx" event={"ID":"5906fa26-ee75-4260-aecf-32245c75dd17","Type":"ContainerStarted","Data":"868e1d678c4889b94626e086f9dcbd6745639745583b02c4f0caa2225880e32b"}
Jul 16 22:27:21.094658 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:21.094619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx" event={"ID":"5906fa26-ee75-4260-aecf-32245c75dd17","Type":"ContainerStarted","Data":"66493e04628aea26a76199cabd13bd468961e58936df84bb6eaf701e25073e37"}
Jul 16 22:27:21.114737 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:21.114669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx" podStartSLOduration=1.948056265 podStartE2EDuration="2.114648656s" totalImagesPullingTime="166.592391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:21.113400409 +0000 UTC m=+603.181723467" watchObservedRunningTime="2026-07-16 22:27:21.114648656 +0000 UTC m=+603.182971717"
Jul 16 22:27:29.891364 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:29.891323    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:29.891364 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:29.891375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:29.904025 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:29.903997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:27:30.149054 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:27:30.148970    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx"
Jul 16 22:28:42.961663 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:42.961626    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-647bd46cf4-h7tlr"]
Jul 16 22:28:42.965005 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:42.964987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:42.973541 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:42.973520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-647bd46cf4-h7tlr"]
Jul 16 22:28:43.092097 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.092067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jds5g\" (UniqueName: \"kubernetes.io/projected/38787a79-6938-4f12-ba3d-044c2659e025-kube-api-access-jds5g\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.092305 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.092179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38787a79-6938-4f12-ba3d-044c2659e025-tls-cert\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.193217 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.193184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38787a79-6938-4f12-ba3d-044c2659e025-tls-cert\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.193364 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.193225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jds5g\" (UniqueName: \"kubernetes.io/projected/38787a79-6938-4f12-ba3d-044c2659e025-kube-api-access-jds5g\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.195558 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.195536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/38787a79-6938-4f12-ba3d-044c2659e025-tls-cert\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.201074 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.201050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jds5g\" (UniqueName: \"kubernetes.io/projected/38787a79-6938-4f12-ba3d-044c2659e025-kube-api-access-jds5g\") pod \"authorino-647bd46cf4-h7tlr\" (UID: \"38787a79-6938-4f12-ba3d-044c2659e025\") " pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.274350 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.274299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-647bd46cf4-h7tlr"
Jul 16 22:28:43.406562 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.406532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-647bd46cf4-h7tlr"]
Jul 16 22:28:43.409159 ip-10-0-141-9 kubenswrapper[2579]: W0716 22:28:43.409133    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38787a79_6938_4f12_ba3d_044c2659e025.slice/crio-156185ccbae4ee9c39af2a3de0528817cf2146e4d2545102e3216185f6be6ac5 WatchSource:0}: Error finding container 156185ccbae4ee9c39af2a3de0528817cf2146e4d2545102e3216185f6be6ac5: Status 404 returned error can't find the container with id 156185ccbae4ee9c39af2a3de0528817cf2146e4d2545102e3216185f6be6ac5
Jul 16 22:28:43.470829 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:43.470801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647bd46cf4-h7tlr" event={"ID":"38787a79-6938-4f12-ba3d-044c2659e025","Type":"ContainerStarted","Data":"156185ccbae4ee9c39af2a3de0528817cf2146e4d2545102e3216185f6be6ac5"}
Jul 16 22:28:44.482964 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.482781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-647bd46cf4-h7tlr" event={"ID":"38787a79-6938-4f12-ba3d-044c2659e025","Type":"ContainerStarted","Data":"91e1ce71f97683b56d795ceec59ff1ed6310597f15bdb45a3de0829a3633ba65"}
Jul 16 22:28:44.502915 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.502865    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-647bd46cf4-h7tlr" podStartSLOduration=2.149753895 podStartE2EDuration="2.502850667s" totalImagesPullingTime="353.096772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:44.501198416 +0000 UTC m=+686.569521471" watchObservedRunningTime="2026-07-16 22:28:44.502850667 +0000 UTC m=+686.571173721"
Jul 16 22:28:44.537336 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.537297    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:28:44.537941 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.537562    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" podUID="195b1361-63b8-4e04-a6eb-1287a1f89131" containerName="authorino" containerID="cri-o://4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b" gracePeriod=30
Jul 16 22:28:44.786907 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.786877    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:28:44.807475 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.807443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert\") pod \"195b1361-63b8-4e04-a6eb-1287a1f89131\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") "
Jul 16 22:28:44.807658 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.807482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdnxh\" (UniqueName: \"kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh\") pod \"195b1361-63b8-4e04-a6eb-1287a1f89131\" (UID: \"195b1361-63b8-4e04-a6eb-1287a1f89131\") "
Jul 16 22:28:44.810110 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.810078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh" (OuterVolumeSpecName: "kube-api-access-sdnxh") pod "195b1361-63b8-4e04-a6eb-1287a1f89131" (UID: "195b1361-63b8-4e04-a6eb-1287a1f89131"). InnerVolumeSpecName "kube-api-access-sdnxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:28:44.818271 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.818245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "195b1361-63b8-4e04-a6eb-1287a1f89131" (UID: "195b1361-63b8-4e04-a6eb-1287a1f89131"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:28:44.908641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.908603    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/195b1361-63b8-4e04-a6eb-1287a1f89131-tls-cert\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:28:44.908641 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:44.908644    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnxh\" (UniqueName: \"kubernetes.io/projected/195b1361-63b8-4e04-a6eb-1287a1f89131-kube-api-access-sdnxh\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 22:28:45.488835 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.488744    2579 generic.go:356] "Generic (PLEG): container finished" podID="195b1361-63b8-4e04-a6eb-1287a1f89131" containerID="4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b" exitCode=0
Jul 16 22:28:45.488835 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.488788    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79c66c87dc-kz7t2"
Jul 16 22:28:45.489307 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.488828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" event={"ID":"195b1361-63b8-4e04-a6eb-1287a1f89131","Type":"ContainerDied","Data":"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"}
Jul 16 22:28:45.489307 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.488874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79c66c87dc-kz7t2" event={"ID":"195b1361-63b8-4e04-a6eb-1287a1f89131","Type":"ContainerDied","Data":"ff5f96b9d05e1245e806c84fbd7af46108f88d35e7a32dc697761b3b3e5271e3"}
Jul 16 22:28:45.489307 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.488897    2579 scope.go:117] "RemoveContainer" containerID="4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"
Jul 16 22:28:45.499662 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.499622    2579 scope.go:117] "RemoveContainer" containerID="4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"
Jul 16 22:28:45.499980 ip-10-0-141-9 kubenswrapper[2579]: E0716 22:28:45.499962    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b\": container with ID starting with 4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b not found: ID does not exist" containerID="4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"
Jul 16 22:28:45.500059 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.499991    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b"} err="failed to get container status \"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b\": rpc error: code = NotFound desc = could not find container \"4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b\": container with ID starting with 4525a4c3d76fc19a16f81e883956bad08e22b6a463dbd0875b90151767f8c03b not found: ID does not exist"
Jul 16 22:28:45.508957 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.508934    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:28:45.514231 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:45.514207    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79c66c87dc-kz7t2"]
Jul 16 22:28:46.461768 ip-10-0-141-9 kubenswrapper[2579]: I0716 22:28:46.461741    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195b1361-63b8-4e04-a6eb-1287a1f89131" path="/var/lib/kubelet/pods/195b1361-63b8-4e04-a6eb-1287a1f89131/volumes"
Jul 16 23:00:09.569508 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.569474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:00:09.570106 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.569860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195b1361-63b8-4e04-a6eb-1287a1f89131" containerName="authorino"
Jul 16 23:00:09.570106 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.569872    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="195b1361-63b8-4e04-a6eb-1287a1f89131" containerName="authorino"
Jul 16 23:00:09.570106 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.569956    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="195b1361-63b8-4e04-a6eb-1287a1f89131" containerName="authorino"
Jul 16 23:00:09.573276 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.573256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.576046 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.576020    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25067845\"/\"kube-root-ca.crt\""
Jul 16 23:00:09.576167 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.576025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-25067845\"/\"openshift-service-ca.crt\""
Jul 16 23:00:09.576647 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.576629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25067845\"/\"e2e-lifecycle-model-25067845-kserve-self-signed-certs\""
Jul 16 23:00:09.576903 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.576888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-25067845\"/\"default-dockercfg-c4ggm\""
Jul 16 23:00:09.584923 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.584903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:00:09.634473 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.634637 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634482    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.634637 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87s8\" (UniqueName: \"kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.634799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.634799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.634799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.634764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.735867 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.735837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.735884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.735912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87s8\" (UniqueName: \"kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.735974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.735997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.736036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736360 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.736335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736501 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.736480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.736564 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.736515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.738101 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.738083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.738443 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.738422    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.748195 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.748167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87s8\" (UniqueName: \"kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8\") pod \"e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") " pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:09.883411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:09.883331    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:10.015446 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:10.015412    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:00:10.018103 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:00:10.018079    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6333799e_ea1f_4468_bc17_1928b7456bb3.slice/crio-67267e6e5dd94cfe251003fff1c41155da36528d91a8456ef190f3652d721827 WatchSource:0}: Error finding container 67267e6e5dd94cfe251003fff1c41155da36528d91a8456ef190f3652d721827: Status 404 returned error can't find the container with id 67267e6e5dd94cfe251003fff1c41155da36528d91a8456ef190f3652d721827
Jul 16 23:00:10.860474 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:10.860439    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" event={"ID":"6333799e-ea1f-4468-bc17-1928b7456bb3","Type":"ContainerStarted","Data":"86e7326f6bba0c8cc8b50792f8cfc2bfbdd47b73cc99ad57195e85d8f70f4866"}
Jul 16 23:00:10.860864 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:10.860484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" event={"ID":"6333799e-ea1f-4468-bc17-1928b7456bb3","Type":"ContainerStarted","Data":"67267e6e5dd94cfe251003fff1c41155da36528d91a8456ef190f3652d721827"}
Jul 16 23:00:10.891980 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:10.891932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" podStartSLOduration=1.891916921 podStartE2EDuration="1.891916921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:00:10.890737372 +0000 UTC m=+2572.959060426" watchObservedRunningTime="2026-07-16 23:00:10.891916921 +0000 UTC m=+2572.960239975"
Jul 16 23:00:19.883976 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:19.883943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:19.883976 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:19.883985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:19.896344 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:19.896320    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:19.916835 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:19.916811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:00:46.851481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:00:46.851383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:01:01.872840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:01.872795    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" podUID="6333799e-ea1f-4468-bc17-1928b7456bb3" containerName="main" containerID="cri-o://86e7326f6bba0c8cc8b50792f8cfc2bfbdd47b73cc99ad57195e85d8f70f4866" gracePeriod=45
Jul 16 23:01:02.095941 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.095907    2579 generic.go:356] "Generic (PLEG): container finished" podID="6333799e-ea1f-4468-bc17-1928b7456bb3" containerID="86e7326f6bba0c8cc8b50792f8cfc2bfbdd47b73cc99ad57195e85d8f70f4866" exitCode=0
Jul 16 23:01:02.096093 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.095986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" event={"ID":"6333799e-ea1f-4468-bc17-1928b7456bb3","Type":"ContainerDied","Data":"86e7326f6bba0c8cc8b50792f8cfc2bfbdd47b73cc99ad57195e85d8f70f4866"}
Jul 16 23:01:02.123865 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.123815    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:01:02.179423 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179574 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179574 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179574 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179574 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l87s8\" (UniqueName: \"kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179804 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache\") pod \"6333799e-ea1f-4468-bc17-1928b7456bb3\" (UID: \"6333799e-ea1f-4468-bc17-1928b7456bb3\") "
Jul 16 23:01:02.179804 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:01:02.179804 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home" (OuterVolumeSpecName: "home") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:01:02.179978 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:01:02.180026 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179976    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-tmp-dir\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:02.180026 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.179995    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-home\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:02.181943 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.181895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm" (OuterVolumeSpecName: "dshm") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:01:02.182058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.181987    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8" (OuterVolumeSpecName: "kube-api-access-l87s8") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "kube-api-access-l87s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:01:02.182058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.182006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6333799e-ea1f-4468-bc17-1928b7456bb3" (UID: "6333799e-ea1f-4468-bc17-1928b7456bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:01:02.280742 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.280689    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-model-cache\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:02.280742 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.280740    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6333799e-ea1f-4468-bc17-1928b7456bb3-tls-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:02.280914 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.280751    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6333799e-ea1f-4468-bc17-1928b7456bb3-dshm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:02.280914 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:02.280765    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87s8\" (UniqueName: \"kubernetes.io/projected/6333799e-ea1f-4468-bc17-1928b7456bb3-kube-api-access-l87s8\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:01:03.101835 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:03.101801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz" event={"ID":"6333799e-ea1f-4468-bc17-1928b7456bb3","Type":"ContainerDied","Data":"67267e6e5dd94cfe251003fff1c41155da36528d91a8456ef190f3652d721827"}
Jul 16 23:01:03.102240 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:03.101821    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"
Jul 16 23:01:03.102240 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:03.101851    2579 scope.go:117] "RemoveContainer" containerID="86e7326f6bba0c8cc8b50792f8cfc2bfbdd47b73cc99ad57195e85d8f70f4866"
Jul 16 23:01:03.120299 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:03.120270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:01:03.125125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:03.125102    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-25067845/e2e-lifecycle-model-25067845-kserve-665fd6b755-wr2gz"]
Jul 16 23:01:04.461519 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:01:04.461490    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6333799e-ea1f-4468-bc17-1928b7456bb3" path="/var/lib/kubelet/pods/6333799e-ea1f-4468-bc17-1928b7456bb3/volumes"
Jul 16 23:02:40.440544 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.440511    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:02:40.441092 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.441069    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6333799e-ea1f-4468-bc17-1928b7456bb3" containerName="main"
Jul 16 23:02:40.441092 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.441088    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6333799e-ea1f-4468-bc17-1928b7456bb3" containerName="main"
Jul 16 23:02:40.441211 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.441185    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6333799e-ea1f-4468-bc17-1928b7456bb3" containerName="main"
Jul 16 23:02:40.444164 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.444145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:02:40.446446 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.446426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:02:40.451359 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.451336    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:02:40.495404 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.495370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm\" (UID: \"35af5bba-4741-4b71-ad50-45e217d2c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:02:40.595726 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.595683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm\" (UID: \"35af5bba-4741-4b71-ad50-45e217d2c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:02:40.596259 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.596229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm\" (UID: \"35af5bba-4741-4b71-ad50-45e217d2c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:02:40.754825 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.754756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:02:40.882634 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.882611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:02:40.884915 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:02:40.884889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35af5bba_4741_4b71_ad50_45e217d2c790.slice/crio-edc70299ed3a40e9beb0bc0c33d5437b67ce18aea7226486360aaf2c9fdb1be5 WatchSource:0}: Error finding container edc70299ed3a40e9beb0bc0c33d5437b67ce18aea7226486360aaf2c9fdb1be5: Status 404 returned error can't find the container with id edc70299ed3a40e9beb0bc0c33d5437b67ce18aea7226486360aaf2c9fdb1be5
Jul 16 23:02:40.886948 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:40.886857    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:02:41.542688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:41.542658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerStarted","Data":"edc70299ed3a40e9beb0bc0c33d5437b67ce18aea7226486360aaf2c9fdb1be5"}
Jul 16 23:02:43.560039 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:43.560010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerStarted","Data":"c263f3b60fec6beca18d106b6cf9f19392e8569cf59660e38e560da200ca0600"}
Jul 16 23:02:43.575994 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:43.575952    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" podStartSLOduration=1.250565224 podStartE2EDuration="3.575936762s" totalImagesPullingTime="2.325371538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:02:43.574899429 +0000 UTC m=+2725.643222479" watchObservedRunningTime="2026-07-16 23:02:43.575936762 +0000 UTC m=+2725.644259816"
Jul 16 23:02:44.565951 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:44.565922    2579 generic.go:356] "Generic (PLEG): container finished" podID="35af5bba-4741-4b71-ad50-45e217d2c790" containerID="c263f3b60fec6beca18d106b6cf9f19392e8569cf59660e38e560da200ca0600" exitCode=6
Jul 16 23:02:44.566395 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:44.565978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerDied","Data":"c263f3b60fec6beca18d106b6cf9f19392e8569cf59660e38e560da200ca0600"}
Jul 16 23:02:44.566395 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:44.566243    2579 scope.go:117] "RemoveContainer" containerID="c263f3b60fec6beca18d106b6cf9f19392e8569cf59660e38e560da200ca0600"
Jul 16 23:02:45.572180 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:45.572151    2579 generic.go:356] "Generic (PLEG): container finished" podID="35af5bba-4741-4b71-ad50-45e217d2c790" containerID="4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340" exitCode=6
Jul 16 23:02:45.572621 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:45.572233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerDied","Data":"4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340"}
Jul 16 23:02:45.572621 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:45.572303    2579 scope.go:117] "RemoveContainer" containerID="c263f3b60fec6beca18d106b6cf9f19392e8569cf59660e38e560da200ca0600"
Jul 16 23:02:45.572621 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:45.572525    2579 scope.go:117] "RemoveContainer" containerID="4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340"
Jul 16 23:02:45.572786 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:02:45.572726    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm_odh-ai-gateway-infra(35af5bba-4741-4b71-ad50-45e217d2c790)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" podUID="35af5bba-4741-4b71-ad50-45e217d2c790"
Jul 16 23:02:46.578394 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:46.578369    2579 scope.go:117] "RemoveContainer" containerID="4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340"
Jul 16 23:02:46.578774 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:02:46.578553    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm_odh-ai-gateway-infra(35af5bba-4741-4b71-ad50-45e217d2c790)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" podUID="35af5bba-4741-4b71-ad50-45e217d2c790"
Jul 16 23:02:58.973232 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:58.973191    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:02:58.976541 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:58.976508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:02:58.986272 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:58.986245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:02:59.037799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.037772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-2k8hf\" (UID: \"4e1205fc-70e6-4ccd-8901-df6778512529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:02:59.138675 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.138640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-2k8hf\" (UID: \"4e1205fc-70e6-4ccd-8901-df6778512529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:02:59.139234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.139216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-2k8hf\" (UID: \"4e1205fc-70e6-4ccd-8901-df6778512529\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:02:59.287357 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.287304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:02:59.414634 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.414611    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:02:59.416482 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:02:59.416457    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1205fc_70e6_4ccd_8901_df6778512529.slice/crio-86b4f956929cee706ae71f778122e1467bc6b5e1d2084beb437dc8b0fcc8ba7e WatchSource:0}: Error finding container 86b4f956929cee706ae71f778122e1467bc6b5e1d2084beb437dc8b0fcc8ba7e: Status 404 returned error can't find the container with id 86b4f956929cee706ae71f778122e1467bc6b5e1d2084beb437dc8b0fcc8ba7e
Jul 16 23:02:59.456098 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.456077    2579 scope.go:117] "RemoveContainer" containerID="4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340"
Jul 16 23:02:59.636928 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:02:59.636899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" event={"ID":"4e1205fc-70e6-4ccd-8901-df6778512529","Type":"ContainerStarted","Data":"86b4f956929cee706ae71f778122e1467bc6b5e1d2084beb437dc8b0fcc8ba7e"}
Jul 16 23:03:00.642861 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.642833    2579 generic.go:356] "Generic (PLEG): container finished" podID="35af5bba-4741-4b71-ad50-45e217d2c790" containerID="0eb59f6401950df7e6f4b6afc13c689eb9b2320abfe6b655928aa1d1e9d2da6b" exitCode=6
Jul 16 23:03:00.643320 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.642905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerDied","Data":"0eb59f6401950df7e6f4b6afc13c689eb9b2320abfe6b655928aa1d1e9d2da6b"}
Jul 16 23:03:00.643320 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.642975    2579 scope.go:117] "RemoveContainer" containerID="4cc9ded80f4fd085ba5d3bb745ff465052e079346cbecfc7204097c4b14f7340"
Jul 16 23:03:00.643320 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.643261    2579 scope.go:117] "RemoveContainer" containerID="0eb59f6401950df7e6f4b6afc13c689eb9b2320abfe6b655928aa1d1e9d2da6b"
Jul 16 23:03:00.643502 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:00.643474    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm_odh-ai-gateway-infra(35af5bba-4741-4b71-ad50-45e217d2c790)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" podUID="35af5bba-4741-4b71-ad50-45e217d2c790"
Jul 16 23:03:00.644496 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.644476    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e1205fc-70e6-4ccd-8901-df6778512529" containerID="1df842268c0b44708ecc2ae69dcce6861b61b2d17ec34d4c86f513666dc24b9e" exitCode=6
Jul 16 23:03:00.644594 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.644524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" event={"ID":"4e1205fc-70e6-4ccd-8901-df6778512529","Type":"ContainerDied","Data":"1df842268c0b44708ecc2ae69dcce6861b61b2d17ec34d4c86f513666dc24b9e"}
Jul 16 23:03:00.644745 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:00.644731    2579 scope.go:117] "RemoveContainer" containerID="1df842268c0b44708ecc2ae69dcce6861b61b2d17ec34d4c86f513666dc24b9e"
Jul 16 23:03:01.651372 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.651335    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e1205fc-70e6-4ccd-8901-df6778512529" containerID="e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976" exitCode=6
Jul 16 23:03:01.651830 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.651430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" event={"ID":"4e1205fc-70e6-4ccd-8901-df6778512529","Type":"ContainerDied","Data":"e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976"}
Jul 16 23:03:01.651830 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.651490    2579 scope.go:117] "RemoveContainer" containerID="1df842268c0b44708ecc2ae69dcce6861b61b2d17ec34d4c86f513666dc24b9e"
Jul 16 23:03:01.651830 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.651626    2579 scope.go:117] "RemoveContainer" containerID="e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976"
Jul 16 23:03:01.651983 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:01.651903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-2k8hf_odh-ai-gateway-infra(4e1205fc-70e6-4ccd-8901-df6778512529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" podUID="4e1205fc-70e6-4ccd-8901-df6778512529"
Jul 16 23:03:01.671035 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.671010    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:03:01.805520 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.805498    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:03:01.858656 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.858628    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca\") pod \"35af5bba-4741-4b71-ad50-45e217d2c790\" (UID: \"35af5bba-4741-4b71-ad50-45e217d2c790\") "
Jul 16 23:03:01.859003 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.858978    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35af5bba-4741-4b71-ad50-45e217d2c790" (UID: "35af5bba-4741-4b71-ad50-45e217d2c790"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:03:01.959559 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:01.959496    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35af5bba-4741-4b71-ad50-45e217d2c790-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:03:02.658773 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.658747    2579 scope.go:117] "RemoveContainer" containerID="e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976"
Jul 16 23:03:02.659169 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:02.658942    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-2k8hf_odh-ai-gateway-infra(4e1205fc-70e6-4ccd-8901-df6778512529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" podUID="4e1205fc-70e6-4ccd-8901-df6778512529"
Jul 16 23:03:02.659833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.659805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm" event={"ID":"35af5bba-4741-4b71-ad50-45e217d2c790","Type":"ContainerDied","Data":"edc70299ed3a40e9beb0bc0c33d5437b67ce18aea7226486360aaf2c9fdb1be5"}
Jul 16 23:03:02.659833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.659826    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"
Jul 16 23:03:02.660005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.659868    2579 scope.go:117] "RemoveContainer" containerID="0eb59f6401950df7e6f4b6afc13c689eb9b2320abfe6b655928aa1d1e9d2da6b"
Jul 16 23:03:02.689297 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.689273    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:03:02.692808 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:02.692785    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nkvnm"]
Jul 16 23:03:04.462184 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:04.462151    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" path="/var/lib/kubelet/pods/35af5bba-4741-4b71-ad50-45e217d2c790/volumes"
Jul 16 23:03:15.123972 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.123923    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124295    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124313    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124318    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124330    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124336    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124389    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.124411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.124396    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:15.128836 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.128814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.132333 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.132303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf84b0\"/\"test-model-bf84b0-kserve-self-signed-certs\""
Jul 16 23:03:15.133058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.133026    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf84b0\"/\"openshift-service-ca.crt\""
Jul 16 23:03:15.133190 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.133065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bf84b0\"/\"default-dockercfg-lpzfn\""
Jul 16 23:03:15.133448 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.133423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bf84b0\"/\"kube-root-ca.crt\""
Jul 16 23:03:15.154553 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.154520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:03:15.271319 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.271433 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.271475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.271475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.271544 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vllk8\" (UniqueName: \"kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.271544 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.271510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372375 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372661 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vllk8\" (UniqueName: \"kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372876 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372960 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.372960 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.372861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.374781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.374730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.374877 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.374861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.380910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.380887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vllk8\" (UniqueName: \"kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8\") pod \"test-model-bf84b0-kserve-7f6d55cccc-f7dd4\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") " pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.438674 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.438651    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:15.457188 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.457168    2579 scope.go:117] "RemoveContainer" containerID="e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976"
Jul 16 23:03:15.571056 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.571031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:03:15.572913 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:03:15.572890    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod942da114_bb05_43ed_a03f_ce0f97eed34c.slice/crio-15d8a922281a49ea8957afddbbd93f791933347806484fc2c74fd781f9cd579c WatchSource:0}: Error finding container 15d8a922281a49ea8957afddbbd93f791933347806484fc2c74fd781f9cd579c: Status 404 returned error can't find the container with id 15d8a922281a49ea8957afddbbd93f791933347806484fc2c74fd781f9cd579c
Jul 16 23:03:15.722666 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.722593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" event={"ID":"942da114-bb05-43ed-a03f-ce0f97eed34c","Type":"ContainerStarted","Data":"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"}
Jul 16 23:03:15.722666 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.722639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" event={"ID":"942da114-bb05-43ed-a03f-ce0f97eed34c","Type":"ContainerStarted","Data":"15d8a922281a49ea8957afddbbd93f791933347806484fc2c74fd781f9cd579c"}
Jul 16 23:03:15.741890 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:15.741845    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" podStartSLOduration=0.741830525 podStartE2EDuration="741.830525ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:03:15.740669858 +0000 UTC m=+2757.808992923" watchObservedRunningTime="2026-07-16 23:03:15.741830525 +0000 UTC m=+2757.810153579"
Jul 16 23:03:16.729110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:16.729044    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e1205fc-70e6-4ccd-8901-df6778512529" containerID="be963b06d9f6663d4f65cc69b8787588e4a57d62862abbf4fe3c67778a7658ca" exitCode=6
Jul 16 23:03:16.729471 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:16.729122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" event={"ID":"4e1205fc-70e6-4ccd-8901-df6778512529","Type":"ContainerDied","Data":"be963b06d9f6663d4f65cc69b8787588e4a57d62862abbf4fe3c67778a7658ca"}
Jul 16 23:03:16.729471 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:16.729176    2579 scope.go:117] "RemoveContainer" containerID="e546b3047f5c1704192accdb513d6f76cacc2eadf04508a939fd118836a09976"
Jul 16 23:03:16.729572 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:16.729560    2579 scope.go:117] "RemoveContainer" containerID="be963b06d9f6663d4f65cc69b8787588e4a57d62862abbf4fe3c67778a7658ca"
Jul 16 23:03:16.729797 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:16.729780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-2k8hf_odh-ai-gateway-infra(4e1205fc-70e6-4ccd-8901-df6778512529)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" podUID="4e1205fc-70e6-4ccd-8901-df6778512529"
Jul 16 23:03:17.757212 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:17.757179    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:03:17.888561 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:17.888530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:03:17.994962 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:17.994929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca\") pod \"4e1205fc-70e6-4ccd-8901-df6778512529\" (UID: \"4e1205fc-70e6-4ccd-8901-df6778512529\") "
Jul 16 23:03:17.995196 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:17.995175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e1205fc-70e6-4ccd-8901-df6778512529" (UID: "4e1205fc-70e6-4ccd-8901-df6778512529"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:03:18.095953 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.095933    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e1205fc-70e6-4ccd-8901-df6778512529-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:03:18.740538 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.740470    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"
Jul 16 23:03:18.740676 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.740472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf" event={"ID":"4e1205fc-70e6-4ccd-8901-df6778512529","Type":"ContainerDied","Data":"86b4f956929cee706ae71f778122e1467bc6b5e1d2084beb437dc8b0fcc8ba7e"}
Jul 16 23:03:18.740676 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.740602    2579 scope.go:117] "RemoveContainer" containerID="be963b06d9f6663d4f65cc69b8787588e4a57d62862abbf4fe3c67778a7658ca"
Jul 16 23:03:18.756602 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.756576    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:03:18.761079 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:18.761056    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-2k8hf"]
Jul 16 23:03:20.462249 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:20.462216    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" path="/var/lib/kubelet/pods/4e1205fc-70e6-4ccd-8901-df6778512529/volumes"
Jul 16 23:03:25.439098 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:25.439067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:25.439546 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:25.439112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:25.451108 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:25.451084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:25.783031 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:25.782957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:03:40.480039 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.479971    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480294    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480306    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480322    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480328    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480384    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480394    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="35af5bba-4741-4b71-ad50-45e217d2c790" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480400    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.482337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.480405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:03:40.483154 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.483139    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:03:40.485732 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.485696    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:03:40.490896 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.490870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:03:40.582366 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.582340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg\" (UID: \"a616ce81-7663-4072-b05a-497d12fdc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:03:40.683525 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.683501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg\" (UID: \"a616ce81-7663-4072-b05a-497d12fdc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:03:40.684120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.684096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg\" (UID: \"a616ce81-7663-4072-b05a-497d12fdc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:03:40.794098 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.794037    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:03:40.922674 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:40.922651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:03:40.925086 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:03:40.925060    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda616ce81_7663_4072_b05a_497d12fdc1d9.slice/crio-dbe1d1893e8811757d01995f42aeda65cc92258fbcef09f87bdb2c75cd0fa5ec WatchSource:0}: Error finding container dbe1d1893e8811757d01995f42aeda65cc92258fbcef09f87bdb2c75cd0fa5ec: Status 404 returned error can't find the container with id dbe1d1893e8811757d01995f42aeda65cc92258fbcef09f87bdb2c75cd0fa5ec
Jul 16 23:03:41.842442 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:41.842410    2579 generic.go:356] "Generic (PLEG): container finished" podID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerID="0c2deac5b34bd21ab23eabf42d10fee46b87173c0e54edae47faafc248730947" exitCode=6
Jul 16 23:03:41.842833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:41.842503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerDied","Data":"0c2deac5b34bd21ab23eabf42d10fee46b87173c0e54edae47faafc248730947"}
Jul 16 23:03:41.842833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:41.842557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerStarted","Data":"dbe1d1893e8811757d01995f42aeda65cc92258fbcef09f87bdb2c75cd0fa5ec"}
Jul 16 23:03:41.842833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:41.842659    2579 scope.go:117] "RemoveContainer" containerID="0c2deac5b34bd21ab23eabf42d10fee46b87173c0e54edae47faafc248730947"
Jul 16 23:03:42.848949 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:42.848910    2579 generic.go:356] "Generic (PLEG): container finished" podID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerID="0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5" exitCode=6
Jul 16 23:03:42.849372 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:42.848963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerDied","Data":"0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5"}
Jul 16 23:03:42.849372 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:42.849019    2579 scope.go:117] "RemoveContainer" containerID="0c2deac5b34bd21ab23eabf42d10fee46b87173c0e54edae47faafc248730947"
Jul 16 23:03:42.849372 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:42.849178    2579 scope.go:117] "RemoveContainer" containerID="0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5"
Jul 16 23:03:42.849545 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:42.849390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg_odh-ai-gateway-infra(a616ce81-7663-4072-b05a-497d12fdc1d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9"
Jul 16 23:03:43.854681 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:43.854653    2579 scope.go:117] "RemoveContainer" containerID="0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5"
Jul 16 23:03:43.855091 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:43.854835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg_odh-ai-gateway-infra(a616ce81-7663-4072-b05a-497d12fdc1d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9"
Jul 16 23:03:58.458423 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:58.458397    2579 scope.go:117] "RemoveContainer" containerID="0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5"
Jul 16 23:03:58.928515 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:58.928483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerStarted","Data":"855b92ee9f0e8250c763e65ffecd726ac4016924dffea428b0dd8ea51d14eb68"}
Jul 16 23:03:58.945229 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:58.945189    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" podStartSLOduration=18.945176076 podStartE2EDuration="18.945176076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:03:58.944497633 +0000 UTC m=+2801.012820688" watchObservedRunningTime="2026-07-16 23:03:58.945176076 +0000 UTC m=+2801.013499133"
Jul 16 23:03:59.934628 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:59.934600    2579 generic.go:356] "Generic (PLEG): container finished" podID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerID="855b92ee9f0e8250c763e65ffecd726ac4016924dffea428b0dd8ea51d14eb68" exitCode=6
Jul 16 23:03:59.935095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:59.934667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerDied","Data":"855b92ee9f0e8250c763e65ffecd726ac4016924dffea428b0dd8ea51d14eb68"}
Jul 16 23:03:59.935095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:59.934756    2579 scope.go:117] "RemoveContainer" containerID="0a17685e318403810a93e48943b9b1d912c4d021bf28b5316f9c27eec27686a5"
Jul 16 23:03:59.935095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:59.935024    2579 scope.go:117] "RemoveContainer" containerID="855b92ee9f0e8250c763e65ffecd726ac4016924dffea428b0dd8ea51d14eb68"
Jul 16 23:03:59.935212 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:03:59.935190    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg_odh-ai-gateway-infra(a616ce81-7663-4072-b05a-497d12fdc1d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9"
Jul 16 23:03:59.959427 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:03:59.959403    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:04:01.071126 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.071106    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:04:01.153263 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.153238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca\") pod \"a616ce81-7663-4072-b05a-497d12fdc1d9\" (UID: \"a616ce81-7663-4072-b05a-497d12fdc1d9\") "
Jul 16 23:04:01.153610 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.153589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a616ce81-7663-4072-b05a-497d12fdc1d9" (UID: "a616ce81-7663-4072-b05a-497d12fdc1d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:01.254482 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.254426    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a616ce81-7663-4072-b05a-497d12fdc1d9-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:01.946170 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.946145    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"
Jul 16 23:04:01.946345 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.946180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg" event={"ID":"a616ce81-7663-4072-b05a-497d12fdc1d9","Type":"ContainerDied","Data":"dbe1d1893e8811757d01995f42aeda65cc92258fbcef09f87bdb2c75cd0fa5ec"}
Jul 16 23:04:01.946345 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.946235    2579 scope.go:117] "RemoveContainer" containerID="855b92ee9f0e8250c763e65ffecd726ac4016924dffea428b0dd8ea51d14eb68"
Jul 16 23:04:01.975168 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.975143    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:04:01.983123 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:01.983102    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-5x8mg"]
Jul 16 23:04:02.462185 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:02.462154    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" path="/var/lib/kubelet/pods/a616ce81-7663-4072-b05a-497d12fdc1d9/volumes"
Jul 16 23:04:03.748410 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:03.748375    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:04:17.159675 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.159640    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:17.160154 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160133    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.160234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160156    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.160234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160170    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.160234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160175    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.160234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160192    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:04:17.160234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160198    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1205fc-70e6-4ccd-8901-df6778512529" containerName="revoke-keys"
Jul 16 23:04:17.160411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160291    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.160411 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.160306    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:17.164662 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.164645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:17.166823 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.166806    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:04:17.170156 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.170134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:17.283817 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.283786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb\" (UID: \"6cacfbe5-89f9-4139-b397-4247d46083e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:17.385103 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.385073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb\" (UID: \"6cacfbe5-89f9-4139-b397-4247d46083e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:17.385693 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.385675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb\" (UID: \"6cacfbe5-89f9-4139-b397-4247d46083e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:17.476933 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.476868    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:17.603059 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:17.603035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:17.605683 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:04:17.605657    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cacfbe5_89f9_4139_b397_4247d46083e3.slice/crio-7f7be6088de1507dac49328bed322172bcb66b5bf44236bde9126a3b0b4954eb WatchSource:0}: Error finding container 7f7be6088de1507dac49328bed322172bcb66b5bf44236bde9126a3b0b4954eb: Status 404 returned error can't find the container with id 7f7be6088de1507dac49328bed322172bcb66b5bf44236bde9126a3b0b4954eb
Jul 16 23:04:18.022763 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:18.022733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerStarted","Data":"60f167cf8e4bdef56cfc8bd574b112df8b4ffcfe2df801eb97b0b6c0c4b39a51"}
Jul 16 23:04:18.022890 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:18.022789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerStarted","Data":"7f7be6088de1507dac49328bed322172bcb66b5bf44236bde9126a3b0b4954eb"}
Jul 16 23:04:18.049132 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:18.049079    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" podStartSLOduration=1.0490600429999999 podStartE2EDuration="1.049060043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:04:18.048213268 +0000 UTC m=+2820.116536321" watchObservedRunningTime="2026-07-16 23:04:18.049060043 +0000 UTC m=+2820.117383114"
Jul 16 23:04:18.769393 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:18.769362    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" podUID="942da114-bb05-43ed-a03f-ce0f97eed34c" containerName="main" containerID="cri-o://18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588" gracePeriod=45
Jul 16 23:04:19.010701 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.010681    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:04:19.034140 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.034108    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerID="60f167cf8e4bdef56cfc8bd574b112df8b4ffcfe2df801eb97b0b6c0c4b39a51" exitCode=6
Jul 16 23:04:19.034266 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.034196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerDied","Data":"60f167cf8e4bdef56cfc8bd574b112df8b4ffcfe2df801eb97b0b6c0c4b39a51"}
Jul 16 23:04:19.034352 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.034338    2579 scope.go:117] "RemoveContainer" containerID="60f167cf8e4bdef56cfc8bd574b112df8b4ffcfe2df801eb97b0b6c0c4b39a51"
Jul 16 23:04:19.035468 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.035445    2579 generic.go:356] "Generic (PLEG): container finished" podID="942da114-bb05-43ed-a03f-ce0f97eed34c" containerID="18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588" exitCode=0
Jul 16 23:04:19.035573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.035479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" event={"ID":"942da114-bb05-43ed-a03f-ce0f97eed34c","Type":"ContainerDied","Data":"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"}
Jul 16 23:04:19.035573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.035494    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"
Jul 16 23:04:19.035573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.035508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4" event={"ID":"942da114-bb05-43ed-a03f-ce0f97eed34c","Type":"ContainerDied","Data":"15d8a922281a49ea8957afddbbd93f791933347806484fc2c74fd781f9cd579c"}
Jul 16 23:04:19.035573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.035529    2579 scope.go:117] "RemoveContainer" containerID="18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"
Jul 16 23:04:19.045815 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.045797    2579 scope.go:117] "RemoveContainer" containerID="18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"
Jul 16 23:04:19.046156 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:19.046131    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588\": container with ID starting with 18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588 not found: ID does not exist" containerID="18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"
Jul 16 23:04:19.046236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.046170    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588"} err="failed to get container status \"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588\": rpc error: code = NotFound desc = could not find container \"18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588\": container with ID starting with 18329b7e7f9d612aab513d53be41e39f396978d87e827ea5ad702b570b6c4588 not found: ID does not exist"
Jul 16 23:04:19.100514 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100488    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vllk8\" (UniqueName: \"kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100624 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100624 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm\") pod \"942da114-bb05-43ed-a03f-ce0f97eed34c\" (UID: \"942da114-bb05-43ed-a03f-ce0f97eed34c\") "
Jul 16 23:04:19.100939 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home" (OuterVolumeSpecName: "home") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:19.100992 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100950    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:19.100992 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.100958    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache" (OuterVolumeSpecName: "model-cache") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:19.101082 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.101058    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-home\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.101144 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.101086    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-tmp-dir\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.101144 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.101096    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-model-cache\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.102661 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.102632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm" (OuterVolumeSpecName: "dshm") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:04:19.102661 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.102637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8" (OuterVolumeSpecName: "kube-api-access-vllk8") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "kube-api-access-vllk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:04:19.102807 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.102659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "942da114-bb05-43ed-a03f-ce0f97eed34c" (UID: "942da114-bb05-43ed-a03f-ce0f97eed34c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:04:19.201809 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.201759    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vllk8\" (UniqueName: \"kubernetes.io/projected/942da114-bb05-43ed-a03f-ce0f97eed34c-kube-api-access-vllk8\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.201809 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.201779    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/942da114-bb05-43ed-a03f-ce0f97eed34c-tls-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.201809 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.201788    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/942da114-bb05-43ed-a03f-ce0f97eed34c-dshm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:19.355869 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.355848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:04:19.359532 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:19.359509    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bf84b0/test-model-bf84b0-kserve-7f6d55cccc-f7dd4"]
Jul 16 23:04:20.043072 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:20.043034    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerID="a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d" exitCode=6
Jul 16 23:04:20.043522 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:20.043098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerDied","Data":"a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d"}
Jul 16 23:04:20.043522 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:20.043145    2579 scope.go:117] "RemoveContainer" containerID="60f167cf8e4bdef56cfc8bd574b112df8b4ffcfe2df801eb97b0b6c0c4b39a51"
Jul 16 23:04:20.043522 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:20.043348    2579 scope.go:117] "RemoveContainer" containerID="a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d"
Jul 16 23:04:20.043674 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:20.043574    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb_odh-ai-gateway-infra(6cacfbe5-89f9-4139-b397-4247d46083e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3"
Jul 16 23:04:20.461973 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:20.461941    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942da114-bb05-43ed-a03f-ce0f97eed34c" path="/var/lib/kubelet/pods/942da114-bb05-43ed-a03f-ce0f97eed34c/volumes"
Jul 16 23:04:21.048855 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:21.048833    2579 scope.go:117] "RemoveContainer" containerID="a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d"
Jul 16 23:04:21.049237 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:21.048989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb_odh-ai-gateway-infra(6cacfbe5-89f9-4139-b397-4247d46083e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3"
Jul 16 23:04:30.907001 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.906963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"]
Jul 16 23:04:30.907477 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942da114-bb05-43ed-a03f-ce0f97eed34c" containerName="main"
Jul 16 23:04:30.907549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907480    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="942da114-bb05-43ed-a03f-ce0f97eed34c" containerName="main"
Jul 16 23:04:30.907549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907508    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:30.907549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907518    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:30.907696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907613    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a616ce81-7663-4072-b05a-497d12fdc1d9" containerName="revoke-keys"
Jul 16 23:04:30.907696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.907628    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="942da114-bb05-43ed-a03f-ce0f97eed34c" containerName="main"
Jul 16 23:04:30.910599 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.910577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:30.918237 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:30.918211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"]
Jul 16 23:04:31.092149 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:31.092121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq\" (UID: \"4c5899df-aecf-4da3-9b23-54c988a6f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:31.193091 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:31.193031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq\" (UID: \"4c5899df-aecf-4da3-9b23-54c988a6f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:31.193599 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:31.193580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq\" (UID: \"4c5899df-aecf-4da3-9b23-54c988a6f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:31.222478 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:31.222451    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:31.349476 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:31.349446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"]
Jul 16 23:04:31.353723 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:04:31.353681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5899df_aecf_4da3_9b23_54c988a6f721.slice/crio-63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca WatchSource:0}: Error finding container 63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca: Status 404 returned error can't find the container with id 63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca
Jul 16 23:04:32.097337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.097308    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c5899df-aecf-4da3-9b23-54c988a6f721" containerID="65e883fe7f4ce26309d7da3976faf18e55af260f47abaef3583b7de3ac35bf26" exitCode=0
Jul 16 23:04:32.097628 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.097394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq" event={"ID":"4c5899df-aecf-4da3-9b23-54c988a6f721","Type":"ContainerDied","Data":"65e883fe7f4ce26309d7da3976faf18e55af260f47abaef3583b7de3ac35bf26"}
Jul 16 23:04:32.097628 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.097440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq" event={"ID":"4c5899df-aecf-4da3-9b23-54c988a6f721","Type":"ContainerStarted","Data":"63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca"}
Jul 16 23:04:32.501679 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.501651    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:32.504987 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.504971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:32.511913 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.511884    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:32.604807 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.604782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x\" (UID: \"8f160cd8-7c9d-426a-bf36-916618d874fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:32.706037 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.706011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x\" (UID: \"8f160cd8-7c9d-426a-bf36-916618d874fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:32.706578 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.706559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x\" (UID: \"8f160cd8-7c9d-426a-bf36-916618d874fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:32.816751 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.816732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:32.943373 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:32.943350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:32.945135 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:04:32.945111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f160cd8_7c9d_426a_bf36_916618d874fe.slice/crio-20c42832f357ee962e7df2dc4efc44d1e5abf590a77e9590ea86e3774372f913 WatchSource:0}: Error finding container 20c42832f357ee962e7df2dc4efc44d1e5abf590a77e9590ea86e3774372f913: Status 404 returned error can't find the container with id 20c42832f357ee962e7df2dc4efc44d1e5abf590a77e9590ea86e3774372f913
Jul 16 23:04:33.103360 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:33.103296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" event={"ID":"8f160cd8-7c9d-426a-bf36-916618d874fe","Type":"ContainerStarted","Data":"20c42832f357ee962e7df2dc4efc44d1e5abf590a77e9590ea86e3774372f913"}
Jul 16 23:04:33.226261 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:33.226241    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:33.412379 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:33.412313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca\") pod \"4c5899df-aecf-4da3-9b23-54c988a6f721\" (UID: \"4c5899df-aecf-4da3-9b23-54c988a6f721\") "
Jul 16 23:04:33.412643 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:33.412622    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c5899df-aecf-4da3-9b23-54c988a6f721" (UID: "4c5899df-aecf-4da3-9b23-54c988a6f721"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:33.513331 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:33.513302    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c5899df-aecf-4da3-9b23-54c988a6f721-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:34.109380 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.109356    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq"
Jul 16 23:04:34.109798 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.109406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq" event={"ID":"4c5899df-aecf-4da3-9b23-54c988a6f721","Type":"ContainerDied","Data":"63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca"}
Jul 16 23:04:34.109798 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.109453    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63db17785ebafbd6c9b132b15aa20302af4043fb0185bb19e0337d8a37d04fca"
Jul 16 23:04:34.110955 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.110936    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerID="0de6ae04d1e265a67c484a16519de1a53a654a5d07c553900103ded71c020c99" exitCode=6
Jul 16 23:04:34.111056 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.110963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" event={"ID":"8f160cd8-7c9d-426a-bf36-916618d874fe","Type":"ContainerDied","Data":"0de6ae04d1e265a67c484a16519de1a53a654a5d07c553900103ded71c020c99"}
Jul 16 23:04:34.111213 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:34.111196    2579 scope.go:117] "RemoveContainer" containerID="0de6ae04d1e265a67c484a16519de1a53a654a5d07c553900103ded71c020c99"
Jul 16 23:04:35.117379 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:35.117346    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerID="c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3" exitCode=6
Jul 16 23:04:35.117854 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:35.117427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" event={"ID":"8f160cd8-7c9d-426a-bf36-916618d874fe","Type":"ContainerDied","Data":"c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3"}
Jul 16 23:04:35.117854 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:35.117520    2579 scope.go:117] "RemoveContainer" containerID="0de6ae04d1e265a67c484a16519de1a53a654a5d07c553900103ded71c020c99"
Jul 16 23:04:35.117854 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:35.117606    2579 scope.go:117] "RemoveContainer" containerID="c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3"
Jul 16 23:04:35.117854 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:35.117835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x_odh-ai-gateway-infra(8f160cd8-7c9d-426a-bf36-916618d874fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe"
Jul 16 23:04:35.456193 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:35.456129    2579 scope.go:117] "RemoveContainer" containerID="a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d"
Jul 16 23:04:36.124693 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:36.124670    2579 scope.go:117] "RemoveContainer" containerID="c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3"
Jul 16 23:04:36.125120 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:36.124907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x_odh-ai-gateway-infra(8f160cd8-7c9d-426a-bf36-916618d874fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe"
Jul 16 23:04:36.126571 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:36.126549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerStarted","Data":"5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d"}
Jul 16 23:04:36.126831 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:36.126819    2579 scope.go:117] "RemoveContainer" containerID="5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d"
Jul 16 23:04:36.126989 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:36.126974    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb_odh-ai-gateway-infra(6cacfbe5-89f9-4139-b397-4247d46083e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3"
Jul 16 23:04:37.133409 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:37.133376    2579 generic.go:356] "Generic (PLEG): container finished" podID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerID="5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d" exitCode=6
Jul 16 23:04:37.133905 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:37.133451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerDied","Data":"5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d"}
Jul 16 23:04:37.133905 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:37.133517    2579 scope.go:117] "RemoveContainer" containerID="a90cb946ba1f6ca4b571c5f7e979e5d8b93a41b6d694548921ca30f37347f04d"
Jul 16 23:04:37.133905 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:37.133824    2579 scope.go:117] "RemoveContainer" containerID="5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d"
Jul 16 23:04:37.134075 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:37.134001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb_odh-ai-gateway-infra(6cacfbe5-89f9-4139-b397-4247d46083e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3"
Jul 16 23:04:37.162993 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:37.162968    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:38.270141 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:38.270115    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:38.448915 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:38.448843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca\") pod \"6cacfbe5-89f9-4139-b397-4247d46083e3\" (UID: \"6cacfbe5-89f9-4139-b397-4247d46083e3\") "
Jul 16 23:04:38.449169 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:38.449148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6cacfbe5-89f9-4139-b397-4247d46083e3" (UID: "6cacfbe5-89f9-4139-b397-4247d46083e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:38.549413 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:38.549389    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cacfbe5-89f9-4139-b397-4247d46083e3-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:39.145021 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:39.144995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"
Jul 16 23:04:39.145189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:39.145034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb" event={"ID":"6cacfbe5-89f9-4139-b397-4247d46083e3","Type":"ContainerDied","Data":"7f7be6088de1507dac49328bed322172bcb66b5bf44236bde9126a3b0b4954eb"}
Jul 16 23:04:39.145189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:39.145094    2579 scope.go:117] "RemoveContainer" containerID="5a33181ac1fa3e203ed362c46938ea3f1ff8f79c640ba1ffbe3a439122d2c08d"
Jul 16 23:04:39.162299 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:39.162278    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:39.166488 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:39.166467    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-ss2vb"]
Jul 16 23:04:40.461901 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.461868    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" path="/var/lib/kubelet/pods/6cacfbe5-89f9-4139-b397-4247d46083e3/volumes"
Jul 16 23:04:40.530599 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.530555    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:04:40.530975 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.530961    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5899df-aecf-4da3-9b23-54c988a6f721" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.530977    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5899df-aecf-4da3-9b23-54c988a6f721" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.530994    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531000    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531008    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531013    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531023    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531028    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531083    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531090    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531097    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5899df-aecf-4da3-9b23-54c988a6f721" containerName="revoke-keys"
Jul 16 23:04:40.531235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.531102    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cacfbe5-89f9-4139-b397-4247d46083e3" containerName="revoke-keys"
Jul 16 23:04:40.535259 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.535242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:04:40.540973 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.540939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:04:40.565323 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.565290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd\" (UID: \"20751565-4501-4eac-96a3-e537ccdcdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:04:40.666378 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.666354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd\" (UID: \"20751565-4501-4eac-96a3-e537ccdcdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:04:40.666923 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.666905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd\" (UID: \"20751565-4501-4eac-96a3-e537ccdcdd22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:04:40.847273 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.847253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:04:40.973059 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:40.973038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:04:40.974919 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:04:40.974891    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20751565_4501_4eac_96a3_e537ccdcdd22.slice/crio-3ed3d4794287de7bdd706c1f161334989098811500550cb2da65f1fb9d755f0e WatchSource:0}: Error finding container 3ed3d4794287de7bdd706c1f161334989098811500550cb2da65f1fb9d755f0e: Status 404 returned error can't find the container with id 3ed3d4794287de7bdd706c1f161334989098811500550cb2da65f1fb9d755f0e
Jul 16 23:04:41.155976 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:41.155909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" event={"ID":"20751565-4501-4eac-96a3-e537ccdcdd22","Type":"ContainerStarted","Data":"3ed3d4794287de7bdd706c1f161334989098811500550cb2da65f1fb9d755f0e"}
Jul 16 23:04:42.161724 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:42.161681    2579 generic.go:356] "Generic (PLEG): container finished" podID="20751565-4501-4eac-96a3-e537ccdcdd22" containerID="71ce458ca49c30013b4b5174b9246059d73197e87b042c5b5de899eba1fb4ed3" exitCode=6
Jul 16 23:04:42.162075 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:42.161747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" event={"ID":"20751565-4501-4eac-96a3-e537ccdcdd22","Type":"ContainerDied","Data":"71ce458ca49c30013b4b5174b9246059d73197e87b042c5b5de899eba1fb4ed3"}
Jul 16 23:04:42.162075 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:42.161971    2579 scope.go:117] "RemoveContainer" containerID="71ce458ca49c30013b4b5174b9246059d73197e87b042c5b5de899eba1fb4ed3"
Jul 16 23:04:43.168354 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:43.168273    2579 generic.go:356] "Generic (PLEG): container finished" podID="20751565-4501-4eac-96a3-e537ccdcdd22" containerID="4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077" exitCode=6
Jul 16 23:04:43.168740 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:43.168363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" event={"ID":"20751565-4501-4eac-96a3-e537ccdcdd22","Type":"ContainerDied","Data":"4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077"}
Jul 16 23:04:43.168740 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:43.168448    2579 scope.go:117] "RemoveContainer" containerID="71ce458ca49c30013b4b5174b9246059d73197e87b042c5b5de899eba1fb4ed3"
Jul 16 23:04:43.168740 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:43.168492    2579 scope.go:117] "RemoveContainer" containerID="4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077"
Jul 16 23:04:43.168740 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:43.168694    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd_odh-ai-gateway-infra(20751565-4501-4eac-96a3-e537ccdcdd22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" podUID="20751565-4501-4eac-96a3-e537ccdcdd22"
Jul 16 23:04:44.174841 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:44.174814    2579 scope.go:117] "RemoveContainer" containerID="4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077"
Jul 16 23:04:44.175214 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:44.174999    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd_odh-ai-gateway-infra(20751565-4501-4eac-96a3-e537ccdcdd22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" podUID="20751565-4501-4eac-96a3-e537ccdcdd22"
Jul 16 23:04:47.456726 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:47.456678    2579 scope.go:117] "RemoveContainer" containerID="c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3"
Jul 16 23:04:48.196880 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:48.196844    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerID="4d6636b1a717d7c18e85c7fbb8bff893db0ca35e2f8e9eea7d7cc82c9c02c162" exitCode=6
Jul 16 23:04:48.197051 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:48.196914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" event={"ID":"8f160cd8-7c9d-426a-bf36-916618d874fe","Type":"ContainerDied","Data":"4d6636b1a717d7c18e85c7fbb8bff893db0ca35e2f8e9eea7d7cc82c9c02c162"}
Jul 16 23:04:48.197051 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:48.196985    2579 scope.go:117] "RemoveContainer" containerID="c867fc8ba9f5c60603df79116cddf3b3f483dd32d075d1735bc7f268cb214bc3"
Jul 16 23:04:48.197343 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:48.197327    2579 scope.go:117] "RemoveContainer" containerID="4d6636b1a717d7c18e85c7fbb8bff893db0ca35e2f8e9eea7d7cc82c9c02c162"
Jul 16 23:04:48.197560 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:48.197544    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x_odh-ai-gateway-infra(8f160cd8-7c9d-426a-bf36-916618d874fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe"
Jul 16 23:04:49.224838 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:49.224804    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:49.365833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:49.365810    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:49.431493 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:49.431463    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca\") pod \"8f160cd8-7c9d-426a-bf36-916618d874fe\" (UID: \"8f160cd8-7c9d-426a-bf36-916618d874fe\") "
Jul 16 23:04:49.431822 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:49.431800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f160cd8-7c9d-426a-bf36-916618d874fe" (UID: "8f160cd8-7c9d-426a-bf36-916618d874fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:04:49.532686 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:49.532606    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f160cd8-7c9d-426a-bf36-916618d874fe-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:04:50.208973 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.208946    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"
Jul 16 23:04:50.209149 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.208974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x" event={"ID":"8f160cd8-7c9d-426a-bf36-916618d874fe","Type":"ContainerDied","Data":"20c42832f357ee962e7df2dc4efc44d1e5abf590a77e9590ea86e3774372f913"}
Jul 16 23:04:50.209149 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.209036    2579 scope.go:117] "RemoveContainer" containerID="4d6636b1a717d7c18e85c7fbb8bff893db0ca35e2f8e9eea7d7cc82c9c02c162"
Jul 16 23:04:50.229101 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.229079    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:50.233688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.233665    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-k4s4x"]
Jul 16 23:04:50.461988 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:50.461909    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" path="/var/lib/kubelet/pods/8f160cd8-7c9d-426a-bf36-916618d874fe/volumes"
Jul 16 23:04:58.458998 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:58.458972    2579 scope.go:117] "RemoveContainer" containerID="4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077"
Jul 16 23:04:59.254438 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:59.254407    2579 generic.go:356] "Generic (PLEG): container finished" podID="20751565-4501-4eac-96a3-e537ccdcdd22" containerID="e5c5fc3fed9a4308db030b9256d0a358ea0f9400397d092128487aac9595531d" exitCode=6
Jul 16 23:04:59.254616 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:59.254483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" event={"ID":"20751565-4501-4eac-96a3-e537ccdcdd22","Type":"ContainerDied","Data":"e5c5fc3fed9a4308db030b9256d0a358ea0f9400397d092128487aac9595531d"}
Jul 16 23:04:59.254616 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:59.254573    2579 scope.go:117] "RemoveContainer" containerID="4adbc5157263fb4bf8eaa8b14f800ff5c6797b05d25764c31d24f738200c0077"
Jul 16 23:04:59.254905 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:04:59.254890    2579 scope.go:117] "RemoveContainer" containerID="e5c5fc3fed9a4308db030b9256d0a358ea0f9400397d092128487aac9595531d"
Jul 16 23:04:59.255135 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:04:59.255110    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd_odh-ai-gateway-infra(20751565-4501-4eac-96a3-e537ccdcdd22)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" podUID="20751565-4501-4eac-96a3-e537ccdcdd22"
Jul 16 23:05:00.280281 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:00.280240    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:05:00.410337 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:00.410312    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:05:00.528031 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:00.528002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca\") pod \"20751565-4501-4eac-96a3-e537ccdcdd22\" (UID: \"20751565-4501-4eac-96a3-e537ccdcdd22\") "
Jul 16 23:05:00.528378 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:00.528357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "20751565-4501-4eac-96a3-e537ccdcdd22" (UID: "20751565-4501-4eac-96a3-e537ccdcdd22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:00.629057 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:00.629030    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20751565-4501-4eac-96a3-e537ccdcdd22-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:01.266784 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:01.266750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"
Jul 16 23:05:01.266952 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:01.266747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd" event={"ID":"20751565-4501-4eac-96a3-e537ccdcdd22","Type":"ContainerDied","Data":"3ed3d4794287de7bdd706c1f161334989098811500550cb2da65f1fb9d755f0e"}
Jul 16 23:05:01.266952 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:01.266894    2579 scope.go:117] "RemoveContainer" containerID="e5c5fc3fed9a4308db030b9256d0a358ea0f9400397d092128487aac9595531d"
Jul 16 23:05:01.290224 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:01.290197    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:05:01.292683 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:01.292661    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-g8ccd"]
Jul 16 23:05:02.461727 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:02.461675    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" path="/var/lib/kubelet/pods/20751565-4501-4eac-96a3-e537ccdcdd22/volumes"
Jul 16 23:05:17.212315 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212229    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212592    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212603    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212620    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212626    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212631    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212651    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212655    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212717    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212727    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212734    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212741    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.212820 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.212747    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:17.215785 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.215767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:17.218032 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.218011    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:05:17.223701 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.223677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:17.369298 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.369272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk\" (UID: \"2a1ae5fa-02fd-437a-b766-332709e284b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:17.470274 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.470184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk\" (UID: \"2a1ae5fa-02fd-437a-b766-332709e284b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:17.470878 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.470858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk\" (UID: \"2a1ae5fa-02fd-437a-b766-332709e284b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:17.526638 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.526611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:17.665011 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:17.664994    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:17.667338 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:05:17.667302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1ae5fa_02fd_437a_b766_332709e284b1.slice/crio-f19b65bef6c4009e7766701d935df71304c93ca3c36c140ad9f4fafc919bb24b WatchSource:0}: Error finding container f19b65bef6c4009e7766701d935df71304c93ca3c36c140ad9f4fafc919bb24b: Status 404 returned error can't find the container with id f19b65bef6c4009e7766701d935df71304c93ca3c36c140ad9f4fafc919bb24b
Jul 16 23:05:18.346579 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:18.346555    2579 generic.go:356] "Generic (PLEG): container finished" podID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerID="680ebd69ebb2baff274cb619065e934884b00ffd3e55d9820ce4c3349085092d" exitCode=6
Jul 16 23:05:18.347004 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:18.346636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" event={"ID":"2a1ae5fa-02fd-437a-b766-332709e284b1","Type":"ContainerDied","Data":"680ebd69ebb2baff274cb619065e934884b00ffd3e55d9820ce4c3349085092d"}
Jul 16 23:05:18.347004 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:18.346688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" event={"ID":"2a1ae5fa-02fd-437a-b766-332709e284b1","Type":"ContainerStarted","Data":"f19b65bef6c4009e7766701d935df71304c93ca3c36c140ad9f4fafc919bb24b"}
Jul 16 23:05:18.347004 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:18.346843    2579 scope.go:117] "RemoveContainer" containerID="680ebd69ebb2baff274cb619065e934884b00ffd3e55d9820ce4c3349085092d"
Jul 16 23:05:19.353462 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:19.353431    2579 generic.go:356] "Generic (PLEG): container finished" podID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerID="1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee" exitCode=6
Jul 16 23:05:19.353892 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:19.353520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" event={"ID":"2a1ae5fa-02fd-437a-b766-332709e284b1","Type":"ContainerDied","Data":"1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee"}
Jul 16 23:05:19.353892 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:19.353589    2579 scope.go:117] "RemoveContainer" containerID="680ebd69ebb2baff274cb619065e934884b00ffd3e55d9820ce4c3349085092d"
Jul 16 23:05:19.353892 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:19.353753    2579 scope.go:117] "RemoveContainer" containerID="1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee"
Jul 16 23:05:19.354056 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:19.353982    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk_odh-ai-gateway-infra(2a1ae5fa-02fd-437a-b766-332709e284b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1"
Jul 16 23:05:20.359840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.359814    2579 scope.go:117] "RemoveContainer" containerID="1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee"
Jul 16 23:05:20.360185 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:20.360012    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk_odh-ai-gateway-infra(2a1ae5fa-02fd-437a-b766-332709e284b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1"
Jul 16 23:05:20.794939 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.794861    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"]
Jul 16 23:05:20.795242 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.795230    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:20.795284 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.795244    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:20.795284 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.795255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:20.795284 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.795260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20751565-4501-4eac-96a3-e537ccdcdd22" containerName="revoke-keys"
Jul 16 23:05:20.795381 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.795310    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f160cd8-7c9d-426a-bf36-916618d874fe" containerName="revoke-keys"
Jul 16 23:05:20.802452 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.802427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:20.805987 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.805958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"]
Jul 16 23:05:20.903275 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:20.903233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b9c909-9dphd\" (UID: \"0791022f-f658-42ab-86e1-727188a4255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:21.004236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:21.004201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b9c909-9dphd\" (UID: \"0791022f-f658-42ab-86e1-727188a4255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:21.004843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:21.004822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b9c909-9dphd\" (UID: \"0791022f-f658-42ab-86e1-727188a4255b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:21.116520 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:21.116488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:21.272036 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:21.272013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"]
Jul 16 23:05:21.274449 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:05:21.274423    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0791022f_f658_42ab_86e1_727188a4255b.slice/crio-3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd WatchSource:0}: Error finding container 3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd: Status 404 returned error can't find the container with id 3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd
Jul 16 23:05:21.365364 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:21.365332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd" event={"ID":"0791022f-f658-42ab-86e1-727188a4255b","Type":"ContainerStarted","Data":"3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd"}
Jul 16 23:05:22.370634 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:22.370602    2579 generic.go:356] "Generic (PLEG): container finished" podID="0791022f-f658-42ab-86e1-727188a4255b" containerID="5a11c5f112a98831c0d84f4fb2a78a6772eb151939cfef739c41b874a70077ab" exitCode=0
Jul 16 23:05:22.371044 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:22.370670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd" event={"ID":"0791022f-f658-42ab-86e1-727188a4255b","Type":"ContainerDied","Data":"5a11c5f112a98831c0d84f4fb2a78a6772eb151939cfef739c41b874a70077ab"}
Jul 16 23:05:23.505178 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:23.505151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:23.625237 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:23.625204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca\") pod \"0791022f-f658-42ab-86e1-727188a4255b\" (UID: \"0791022f-f658-42ab-86e1-727188a4255b\") "
Jul 16 23:05:23.625535 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:23.625516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0791022f-f658-42ab-86e1-727188a4255b" (UID: "0791022f-f658-42ab-86e1-727188a4255b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:23.726728 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:23.726653    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0791022f-f658-42ab-86e1-727188a4255b-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:24.383117 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:24.383085    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd"
Jul 16 23:05:24.383304 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:24.383113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b9c909-9dphd" event={"ID":"0791022f-f658-42ab-86e1-727188a4255b","Type":"ContainerDied","Data":"3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd"}
Jul 16 23:05:24.383304 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:24.383167    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3237dc44badf3f30ea60017a5aece74cd8d13d58869cf186af7035b980630ebd"
Jul 16 23:05:29.127051 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.127018    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:29.127428 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.127371    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0791022f-f658-42ab-86e1-727188a4255b" containerName="revoke-keys"
Jul 16 23:05:29.127428 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.127382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0791022f-f658-42ab-86e1-727188a4255b" containerName="revoke-keys"
Jul 16 23:05:29.127501 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.127438    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0791022f-f658-42ab-86e1-727188a4255b" containerName="revoke-keys"
Jul 16 23:05:29.130384 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.130368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:29.138452 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.138424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:29.277576 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.277540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-fw59g\" (UID: \"e1cf7221-7c5d-4c8e-bb77-101c24179316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:29.378542 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.378461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-fw59g\" (UID: \"e1cf7221-7c5d-4c8e-bb77-101c24179316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:29.379055 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.379037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-fw59g\" (UID: \"e1cf7221-7c5d-4c8e-bb77-101c24179316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:29.442539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.442509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:29.571690 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:29.571663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:29.573231 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:05:29.573200    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1cf7221_7c5d_4c8e_bb77_101c24179316.slice/crio-b5e51fcbdef82f9cbeadd7d3fb28ae0d4d40dd4616e736c42a0af0672c9f95ce WatchSource:0}: Error finding container b5e51fcbdef82f9cbeadd7d3fb28ae0d4d40dd4616e736c42a0af0672c9f95ce: Status 404 returned error can't find the container with id b5e51fcbdef82f9cbeadd7d3fb28ae0d4d40dd4616e736c42a0af0672c9f95ce
Jul 16 23:05:30.411917 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:30.411828    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerID="ffb38014c93e0d667c7c85b53927361dc6d885f4c2a14e1e0c5a4ca0021f3be6" exitCode=6
Jul 16 23:05:30.412380 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:30.411915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" event={"ID":"e1cf7221-7c5d-4c8e-bb77-101c24179316","Type":"ContainerDied","Data":"ffb38014c93e0d667c7c85b53927361dc6d885f4c2a14e1e0c5a4ca0021f3be6"}
Jul 16 23:05:30.412380 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:30.411973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" event={"ID":"e1cf7221-7c5d-4c8e-bb77-101c24179316","Type":"ContainerStarted","Data":"b5e51fcbdef82f9cbeadd7d3fb28ae0d4d40dd4616e736c42a0af0672c9f95ce"}
Jul 16 23:05:30.412380 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:30.412083    2579 scope.go:117] "RemoveContainer" containerID="ffb38014c93e0d667c7c85b53927361dc6d885f4c2a14e1e0c5a4ca0021f3be6"
Jul 16 23:05:31.418245 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:31.418164    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerID="53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5" exitCode=6
Jul 16 23:05:31.418649 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:31.418240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" event={"ID":"e1cf7221-7c5d-4c8e-bb77-101c24179316","Type":"ContainerDied","Data":"53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5"}
Jul 16 23:05:31.418649 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:31.418298    2579 scope.go:117] "RemoveContainer" containerID="ffb38014c93e0d667c7c85b53927361dc6d885f4c2a14e1e0c5a4ca0021f3be6"
Jul 16 23:05:31.418649 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:31.418468    2579 scope.go:117] "RemoveContainer" containerID="53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5"
Jul 16 23:05:31.418788 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:31.418692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-fw59g_odh-ai-gateway-infra(e1cf7221-7c5d-4c8e-bb77-101c24179316)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316"
Jul 16 23:05:32.425044 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:32.425016    2579 scope.go:117] "RemoveContainer" containerID="53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5"
Jul 16 23:05:32.425406 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:32.425205    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-fw59g_odh-ai-gateway-infra(e1cf7221-7c5d-4c8e-bb77-101c24179316)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316"
Jul 16 23:05:34.457179 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:34.457144    2579 scope.go:117] "RemoveContainer" containerID="1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee"
Jul 16 23:05:35.440308 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:35.440269    2579 generic.go:356] "Generic (PLEG): container finished" podID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerID="b971380d4f15240fbe9acd8d9117c73791a237c102043725b59b8f9c808e3a87" exitCode=6
Jul 16 23:05:35.440308 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:35.440313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" event={"ID":"2a1ae5fa-02fd-437a-b766-332709e284b1","Type":"ContainerDied","Data":"b971380d4f15240fbe9acd8d9117c73791a237c102043725b59b8f9c808e3a87"}
Jul 16 23:05:35.440534 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:35.440370    2579 scope.go:117] "RemoveContainer" containerID="1c71918957d42a2ce5668cec58a3c2da933e214ec1a7cfaae835f582d8e6b1ee"
Jul 16 23:05:35.440683 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:35.440668    2579 scope.go:117] "RemoveContainer" containerID="b971380d4f15240fbe9acd8d9117c73791a237c102043725b59b8f9c808e3a87"
Jul 16 23:05:35.440912 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:35.440885    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk_odh-ai-gateway-infra(2a1ae5fa-02fd-437a-b766-332709e284b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1"
Jul 16 23:05:36.469661 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:36.469627    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:36.595854 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:36.595831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:36.634852 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:36.634826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca\") pod \"2a1ae5fa-02fd-437a-b766-332709e284b1\" (UID: \"2a1ae5fa-02fd-437a-b766-332709e284b1\") "
Jul 16 23:05:36.635163 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:36.635139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a1ae5fa-02fd-437a-b766-332709e284b1" (UID: "2a1ae5fa-02fd-437a-b766-332709e284b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:36.735436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:36.735358    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1ae5fa-02fd-437a-b766-332709e284b1-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:37.452314 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:37.452286    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"
Jul 16 23:05:37.452490 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:37.452311    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk" event={"ID":"2a1ae5fa-02fd-437a-b766-332709e284b1","Type":"ContainerDied","Data":"f19b65bef6c4009e7766701d935df71304c93ca3c36c140ad9f4fafc919bb24b"}
Jul 16 23:05:37.452490 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:37.452376    2579 scope.go:117] "RemoveContainer" containerID="b971380d4f15240fbe9acd8d9117c73791a237c102043725b59b8f9c808e3a87"
Jul 16 23:05:37.473742 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:37.473699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:37.480247 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:37.480224    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-mh5wk"]
Jul 16 23:05:38.462074 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:38.462040    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" path="/var/lib/kubelet/pods/2a1ae5fa-02fd-437a-b766-332709e284b1/volumes"
Jul 16 23:05:40.815467 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815776    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815789    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815800    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815806    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815821    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815827    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815905    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.815934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.815912    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:40.820056 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.820039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:40.826107 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.826083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:05:40.870985 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.870959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v\" (UID: \"1cc9712b-f2f1-465c-8853-f899cfc1c1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:40.972400 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.972368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v\" (UID: \"1cc9712b-f2f1-465c-8853-f899cfc1c1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:40.972959 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:40.972942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v\" (UID: \"1cc9712b-f2f1-465c-8853-f899cfc1c1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:41.132926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:41.132853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:41.258601 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:41.258577    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:05:41.260607 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:05:41.260578    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc9712b_f2f1_465c_8853_f899cfc1c1ae.slice/crio-cb8215053bee03958fa8f40afd3d9b8a20199453cd87128ab2c83ed2430bbade WatchSource:0}: Error finding container cb8215053bee03958fa8f40afd3d9b8a20199453cd87128ab2c83ed2430bbade: Status 404 returned error can't find the container with id cb8215053bee03958fa8f40afd3d9b8a20199453cd87128ab2c83ed2430bbade
Jul 16 23:05:41.472502 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:41.472472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" event={"ID":"1cc9712b-f2f1-465c-8853-f899cfc1c1ae","Type":"ContainerStarted","Data":"cb8215053bee03958fa8f40afd3d9b8a20199453cd87128ab2c83ed2430bbade"}
Jul 16 23:05:42.478240 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:42.478209    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerID="30cd2075efac6911336dc1427ffe642dcdbf5f1d62717ecca96811f33aecf037" exitCode=6
Jul 16 23:05:42.482751 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:42.478303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" event={"ID":"1cc9712b-f2f1-465c-8853-f899cfc1c1ae","Type":"ContainerDied","Data":"30cd2075efac6911336dc1427ffe642dcdbf5f1d62717ecca96811f33aecf037"}
Jul 16 23:05:42.482751 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:42.479048    2579 scope.go:117] "RemoveContainer" containerID="30cd2075efac6911336dc1427ffe642dcdbf5f1d62717ecca96811f33aecf037"
Jul 16 23:05:43.484849 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:43.484762    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerID="5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581" exitCode=6
Jul 16 23:05:43.485218 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:43.484844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" event={"ID":"1cc9712b-f2f1-465c-8853-f899cfc1c1ae","Type":"ContainerDied","Data":"5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581"}
Jul 16 23:05:43.485218 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:43.484902    2579 scope.go:117] "RemoveContainer" containerID="30cd2075efac6911336dc1427ffe642dcdbf5f1d62717ecca96811f33aecf037"
Jul 16 23:05:43.485218 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:43.485125    2579 scope.go:117] "RemoveContainer" containerID="5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581"
Jul 16 23:05:43.485334 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:43.485317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v_odh-ai-gateway-infra(1cc9712b-f2f1-465c-8853-f899cfc1c1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae"
Jul 16 23:05:44.456304 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:44.456273    2579 scope.go:117] "RemoveContainer" containerID="53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5"
Jul 16 23:05:44.490364 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:44.490343    2579 scope.go:117] "RemoveContainer" containerID="5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581"
Jul 16 23:05:44.490683 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:44.490520    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v_odh-ai-gateway-infra(1cc9712b-f2f1-465c-8853-f899cfc1c1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae"
Jul 16 23:05:45.496573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:45.496492    2579 generic.go:356] "Generic (PLEG): container finished" podID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerID="2da8f6d1066a6ef95554045157ec1ec609bcc86fddc36674146212bb496cab7c" exitCode=6
Jul 16 23:05:45.496573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:45.496567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" event={"ID":"e1cf7221-7c5d-4c8e-bb77-101c24179316","Type":"ContainerDied","Data":"2da8f6d1066a6ef95554045157ec1ec609bcc86fddc36674146212bb496cab7c"}
Jul 16 23:05:45.497028 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:45.496624    2579 scope.go:117] "RemoveContainer" containerID="53f067f4ea5135897013cdb544f1d02a6a38bbe6ea06b69aa13971d066b372e5"
Jul 16 23:05:45.497028 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:45.496940    2579 scope.go:117] "RemoveContainer" containerID="2da8f6d1066a6ef95554045157ec1ec609bcc86fddc36674146212bb496cab7c"
Jul 16 23:05:45.497174 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:45.497135    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-fw59g_odh-ai-gateway-infra(e1cf7221-7c5d-4c8e-bb77-101c24179316)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316"
Jul 16 23:05:46.529480 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.529438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:46.677441 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.677418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:46.720919 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.720888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca\") pod \"e1cf7221-7c5d-4c8e-bb77-101c24179316\" (UID: \"e1cf7221-7c5d-4c8e-bb77-101c24179316\") "
Jul 16 23:05:46.721248 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.721225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1cf7221-7c5d-4c8e-bb77-101c24179316" (UID: "e1cf7221-7c5d-4c8e-bb77-101c24179316"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:46.821500 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.821470    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1cf7221-7c5d-4c8e-bb77-101c24179316-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:46.932421 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932383    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"]
Jul 16 23:05:46.932827 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932809    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932827 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932828    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932855    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932868    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932877    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.932969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932960    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.933267 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:05:46.933267 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.932987    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a1ae5fa-02fd-437a-b766-332709e284b1" containerName="revoke-keys"
Jul 16 23:05:46.935975 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.935956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:46.942273 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:46.942248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"]
Jul 16 23:05:47.023233 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.023198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c14db-svlpb\" (UID: \"635a7f3d-b483-4d4e-bcfd-6788ed8873ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:47.124057 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.123968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c14db-svlpb\" (UID: \"635a7f3d-b483-4d4e-bcfd-6788ed8873ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:47.124610 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.124590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8c14db-svlpb\" (UID: \"635a7f3d-b483-4d4e-bcfd-6788ed8873ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:47.246621 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.246571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:47.381381 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.381359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"]
Jul 16 23:05:47.383290 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:05:47.383255    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a7f3d_b483_4d4e_bcfd_6788ed8873ca.slice/crio-fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63 WatchSource:0}: Error finding container fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63: Status 404 returned error can't find the container with id fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63
Jul 16 23:05:47.509722 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.509685    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"
Jul 16 23:05:47.509916 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.509739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g" event={"ID":"e1cf7221-7c5d-4c8e-bb77-101c24179316","Type":"ContainerDied","Data":"b5e51fcbdef82f9cbeadd7d3fb28ae0d4d40dd4616e736c42a0af0672c9f95ce"}
Jul 16 23:05:47.509916 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.509809    2579 scope.go:117] "RemoveContainer" containerID="2da8f6d1066a6ef95554045157ec1ec609bcc86fddc36674146212bb496cab7c"
Jul 16 23:05:47.511066 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.511037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb" event={"ID":"635a7f3d-b483-4d4e-bcfd-6788ed8873ca","Type":"ContainerStarted","Data":"fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63"}
Jul 16 23:05:47.536188 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.536152    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:47.536592 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:47.536214    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-fw59g"]
Jul 16 23:05:48.462143 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:48.462107    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" path="/var/lib/kubelet/pods/e1cf7221-7c5d-4c8e-bb77-101c24179316/volumes"
Jul 16 23:05:48.517436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:48.517354    2579 generic.go:356] "Generic (PLEG): container finished" podID="635a7f3d-b483-4d4e-bcfd-6788ed8873ca" containerID="b0a0e1cb5f75f2c6aefe9f935853ebf3bfd394726781c5cf69151aa6f9a63258" exitCode=0
Jul 16 23:05:48.517436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:48.517409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb" event={"ID":"635a7f3d-b483-4d4e-bcfd-6788ed8873ca","Type":"ContainerDied","Data":"b0a0e1cb5f75f2c6aefe9f935853ebf3bfd394726781c5cf69151aa6f9a63258"}
Jul 16 23:05:49.652856 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:49.652831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:49.747071 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:49.747038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca\") pod \"635a7f3d-b483-4d4e-bcfd-6788ed8873ca\" (UID: \"635a7f3d-b483-4d4e-bcfd-6788ed8873ca\") "
Jul 16 23:05:49.747387 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:49.747365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "635a7f3d-b483-4d4e-bcfd-6788ed8873ca" (UID: "635a7f3d-b483-4d4e-bcfd-6788ed8873ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:49.848234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:49.848203    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/635a7f3d-b483-4d4e-bcfd-6788ed8873ca-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:50.528173 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:50.528143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb" event={"ID":"635a7f3d-b483-4d4e-bcfd-6788ed8873ca","Type":"ContainerDied","Data":"fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63"}
Jul 16 23:05:50.528173 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:50.528164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8c14db-svlpb"
Jul 16 23:05:50.528384 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:50.528190    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe276bc94f1369fc0df6514fb30875e38c2014dc71c2092b2d5ddb5bb429de63"
Jul 16 23:05:55.457251 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:55.457220    2579 scope.go:117] "RemoveContainer" containerID="5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581"
Jul 16 23:05:56.557845 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:56.557806    2579 generic.go:356] "Generic (PLEG): container finished" podID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerID="f2173ddfc678147a9f47f4c738af0a463bd478ee4376689c1babecffd02cf601" exitCode=6
Jul 16 23:05:56.558245 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:56.557856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" event={"ID":"1cc9712b-f2f1-465c-8853-f899cfc1c1ae","Type":"ContainerDied","Data":"f2173ddfc678147a9f47f4c738af0a463bd478ee4376689c1babecffd02cf601"}
Jul 16 23:05:56.558245 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:56.557911    2579 scope.go:117] "RemoveContainer" containerID="5fdd4e1e195f7da3d85746d7214d2eb78f900ddbaa85bebcea71e983c964d581"
Jul 16 23:05:56.558245 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:56.558172    2579 scope.go:117] "RemoveContainer" containerID="f2173ddfc678147a9f47f4c738af0a463bd478ee4376689c1babecffd02cf601"
Jul 16 23:05:56.558409 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:05:56.558391    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v_odh-ai-gateway-infra(1cc9712b-f2f1-465c-8853-f899cfc1c1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae"
Jul 16 23:05:57.598376 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:57.598342    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:05:57.732345 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:57.732323    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:57.820889 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:57.820858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca\") pod \"1cc9712b-f2f1-465c-8853-f899cfc1c1ae\" (UID: \"1cc9712b-f2f1-465c-8853-f899cfc1c1ae\") "
Jul 16 23:05:57.821202 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:57.821179    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1cc9712b-f2f1-465c-8853-f899cfc1c1ae" (UID: "1cc9712b-f2f1-465c-8853-f899cfc1c1ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:05:57.921793 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:57.921693    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cc9712b-f2f1-465c-8853-f899cfc1c1ae-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:05:58.571235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:58.571199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v" event={"ID":"1cc9712b-f2f1-465c-8853-f899cfc1c1ae","Type":"ContainerDied","Data":"cb8215053bee03958fa8f40afd3d9b8a20199453cd87128ab2c83ed2430bbade"}
Jul 16 23:05:58.571391 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:58.571263    2579 scope.go:117] "RemoveContainer" containerID="f2173ddfc678147a9f47f4c738af0a463bd478ee4376689c1babecffd02cf601"
Jul 16 23:05:58.571391 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:58.571218    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"
Jul 16 23:05:58.586952 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:58.586929    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:05:58.592080 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:05:58.592060    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-nxv8v"]
Jul 16 23:06:00.467029 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:00.466999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" path="/var/lib/kubelet/pods/1cc9712b-f2f1-465c-8853-f899cfc1c1ae/volumes"
Jul 16 23:06:12.595130 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595101    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:12.595619 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595601    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595621    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595637    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595646    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595656    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635a7f3d-b483-4d4e-bcfd-6788ed8873ca" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595664    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635a7f3d-b483-4d4e-bcfd-6788ed8873ca" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595679    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.595725 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595687    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.596077 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595798    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1cf7221-7c5d-4c8e-bb77-101c24179316" containerName="revoke-keys"
Jul 16 23:06:12.596077 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.596077 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595824    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635a7f3d-b483-4d4e-bcfd-6788ed8873ca" containerName="revoke-keys"
Jul 16 23:06:12.596077 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595833    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.596077 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.595843    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc9712b-f2f1-465c-8853-f899cfc1c1ae" containerName="revoke-keys"
Jul 16 23:06:12.600534 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.600515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:12.603041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.603013    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:06:12.606972 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.606948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:12.747040 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.747001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj\" (UID: \"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:12.848549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.848466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj\" (UID: \"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:12.849134 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.849117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj\" (UID: \"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:12.911692 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:12.911663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:13.247474 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:13.247406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:13.249670 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:06:13.249643    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1e7f96_b16e_4ff7_b9cb_b9e2529d7e98.slice/crio-8d46c3d09e123121fcd04bbbadd27cd69f4e1d681386814ae2ec5d394c11fadf WatchSource:0}: Error finding container 8d46c3d09e123121fcd04bbbadd27cd69f4e1d681386814ae2ec5d394c11fadf: Status 404 returned error can't find the container with id 8d46c3d09e123121fcd04bbbadd27cd69f4e1d681386814ae2ec5d394c11fadf
Jul 16 23:06:13.643382 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:13.643349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerStarted","Data":"ec9a9648a9b6b7f2457ba51e7963d423c9549143a47381be8e20ef0cbf598fe5"}
Jul 16 23:06:13.643785 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:13.643395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerStarted","Data":"8d46c3d09e123121fcd04bbbadd27cd69f4e1d681386814ae2ec5d394c11fadf"}
Jul 16 23:06:13.666938 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:13.666891    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" podStartSLOduration=1.666876418 podStartE2EDuration="1.666876418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:13.665169498 +0000 UTC m=+2935.733492551" watchObservedRunningTime="2026-07-16 23:06:13.666876418 +0000 UTC m=+2935.735199527"
Jul 16 23:06:14.650849 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:14.650808    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerID="ec9a9648a9b6b7f2457ba51e7963d423c9549143a47381be8e20ef0cbf598fe5" exitCode=6
Jul 16 23:06:14.651302 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:14.650896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerDied","Data":"ec9a9648a9b6b7f2457ba51e7963d423c9549143a47381be8e20ef0cbf598fe5"}
Jul 16 23:06:14.651302 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:14.651103    2579 scope.go:117] "RemoveContainer" containerID="ec9a9648a9b6b7f2457ba51e7963d423c9549143a47381be8e20ef0cbf598fe5"
Jul 16 23:06:15.657686 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:15.657610    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerID="942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952" exitCode=6
Jul 16 23:06:15.658095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:15.657694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerDied","Data":"942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952"}
Jul 16 23:06:15.658095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:15.657767    2579 scope.go:117] "RemoveContainer" containerID="ec9a9648a9b6b7f2457ba51e7963d423c9549143a47381be8e20ef0cbf598fe5"
Jul 16 23:06:15.658095 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:15.658004    2579 scope.go:117] "RemoveContainer" containerID="942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952"
Jul 16 23:06:15.658287 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:15.658265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj_odh-ai-gateway-infra(7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98"
Jul 16 23:06:16.664456 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:16.664431    2579 scope.go:117] "RemoveContainer" containerID="942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952"
Jul 16 23:06:16.664853 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:16.664606    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj_odh-ai-gateway-infra(7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98"
Jul 16 23:06:17.287162 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.287131    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:17.291783 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.291764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:17.298103 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.298082    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:17.390652 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.390613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq\" (UID: \"ebcad4af-ef69-40da-9268-288f5bc83744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:17.491803 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.491772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq\" (UID: \"ebcad4af-ef69-40da-9268-288f5bc83744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:17.492587 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.492563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq\" (UID: \"ebcad4af-ef69-40da-9268-288f5bc83744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:17.603103 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.603059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:17.738506 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:17.738481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:17.740513 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:06:17.740486    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebcad4af_ef69_40da_9268_288f5bc83744.slice/crio-45160ad73c53dba1e88ae8a816c919d30b2cf99d80eb39baa8e5ef662089a52a WatchSource:0}: Error finding container 45160ad73c53dba1e88ae8a816c919d30b2cf99d80eb39baa8e5ef662089a52a: Status 404 returned error can't find the container with id 45160ad73c53dba1e88ae8a816c919d30b2cf99d80eb39baa8e5ef662089a52a
Jul 16 23:06:18.677696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:18.677614    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebcad4af-ef69-40da-9268-288f5bc83744" containerID="365df040aec1e42394c37e6917da815274a8a7c09cfd23809139c2b42bb74dad" exitCode=6
Jul 16 23:06:18.677696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:18.677653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerDied","Data":"365df040aec1e42394c37e6917da815274a8a7c09cfd23809139c2b42bb74dad"}
Jul 16 23:06:18.677932 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:18.677702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerStarted","Data":"45160ad73c53dba1e88ae8a816c919d30b2cf99d80eb39baa8e5ef662089a52a"}
Jul 16 23:06:18.677991 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:18.677942    2579 scope.go:117] "RemoveContainer" containerID="365df040aec1e42394c37e6917da815274a8a7c09cfd23809139c2b42bb74dad"
Jul 16 23:06:19.684058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:19.683973    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebcad4af-ef69-40da-9268-288f5bc83744" containerID="9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4" exitCode=6
Jul 16 23:06:19.684058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:19.684014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerDied","Data":"9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4"}
Jul 16 23:06:19.684598 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:19.684068    2579 scope.go:117] "RemoveContainer" containerID="365df040aec1e42394c37e6917da815274a8a7c09cfd23809139c2b42bb74dad"
Jul 16 23:06:19.684598 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:19.684256    2579 scope.go:117] "RemoveContainer" containerID="9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4"
Jul 16 23:06:19.684598 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:19.684521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq_odh-ai-gateway-infra(ebcad4af-ef69-40da-9268-288f5bc83744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" podUID="ebcad4af-ef69-40da-9268-288f5bc83744"
Jul 16 23:06:20.690426 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:20.690395    2579 scope.go:117] "RemoveContainer" containerID="9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4"
Jul 16 23:06:20.690817 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:20.690599    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq_odh-ai-gateway-infra(ebcad4af-ef69-40da-9268-288f5bc83744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" podUID="ebcad4af-ef69-40da-9268-288f5bc83744"
Jul 16 23:06:28.460182 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:28.460150    2579 scope.go:117] "RemoveContainer" containerID="942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952"
Jul 16 23:06:28.730312 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:28.730287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerStarted","Data":"6f15c1871abc92a86b7c56ef997523f82cfe0bcb1d3fcb885df1a29b480c62e3"}
Jul 16 23:06:29.737219 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:29.737190    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerID="6f15c1871abc92a86b7c56ef997523f82cfe0bcb1d3fcb885df1a29b480c62e3" exitCode=6
Jul 16 23:06:29.737606 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:29.737227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerDied","Data":"6f15c1871abc92a86b7c56ef997523f82cfe0bcb1d3fcb885df1a29b480c62e3"}
Jul 16 23:06:29.738212 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:29.737275    2579 scope.go:117] "RemoveContainer" containerID="942259c3bf3d6ad84553355e47f8f369827f704201bb28c02936c3fafd082952"
Jul 16 23:06:29.738282 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:29.738272    2579 scope.go:117] "RemoveContainer" containerID="6f15c1871abc92a86b7c56ef997523f82cfe0bcb1d3fcb885df1a29b480c62e3"
Jul 16 23:06:29.738524 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:29.738501    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj_odh-ai-gateway-infra(7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98"
Jul 16 23:06:29.771591 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:29.771561    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:30.852340 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.852309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:30.856938 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.856915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:30.863143 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.863115    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:30.887191 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.887174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:30.915720 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.915679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca\") pod \"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98\" (UID: \"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98\") "
Jul 16 23:06:30.915907 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.915882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp\" (UID: \"32a9eaef-a168-4598-b418-c7bceeb99eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:30.916041 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:30.916008    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" (UID: "7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:31.017120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.017039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp\" (UID: \"32a9eaef-a168-4598-b418-c7bceeb99eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:31.017266 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.017128    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:06:31.017640 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.017620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp\" (UID: \"32a9eaef-a168-4598-b418-c7bceeb99eb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:31.183827 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.183791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:31.517061 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.517034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:31.518649 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:06:31.518623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a9eaef_a168_4598_b418_c7bceeb99eb6.slice/crio-eaffa8f34bd9ac17bf0e49e6fe7c6edaf327b2f0b7a74fbe245cebaeeec6b464 WatchSource:0}: Error finding container eaffa8f34bd9ac17bf0e49e6fe7c6edaf327b2f0b7a74fbe245cebaeeec6b464: Status 404 returned error can't find the container with id eaffa8f34bd9ac17bf0e49e6fe7c6edaf327b2f0b7a74fbe245cebaeeec6b464
Jul 16 23:06:31.748653 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.748618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerStarted","Data":"eaffa8f34bd9ac17bf0e49e6fe7c6edaf327b2f0b7a74fbe245cebaeeec6b464"}
Jul 16 23:06:31.749842 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.749816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj" event={"ID":"7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98","Type":"ContainerDied","Data":"8d46c3d09e123121fcd04bbbadd27cd69f4e1d681386814ae2ec5d394c11fadf"}
Jul 16 23:06:31.749952 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.749869    2579 scope.go:117] "RemoveContainer" containerID="6f15c1871abc92a86b7c56ef997523f82cfe0bcb1d3fcb885df1a29b480c62e3"
Jul 16 23:06:31.749952 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.749831    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"
Jul 16 23:06:31.771615 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.771556    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:31.777376 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:31.777356    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-zgmfj"]
Jul 16 23:06:32.462668 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:32.462585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" path="/var/lib/kubelet/pods/7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98/volumes"
Jul 16 23:06:32.756189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:32.756102    2579 generic.go:356] "Generic (PLEG): container finished" podID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerID="df2ab0487f972c944e31d2894ba376931efa0fd0d7ebe70f591119dd5d6d32d8" exitCode=6
Jul 16 23:06:32.756344 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:32.756196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerDied","Data":"df2ab0487f972c944e31d2894ba376931efa0fd0d7ebe70f591119dd5d6d32d8"}
Jul 16 23:06:32.756404 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:32.756388    2579 scope.go:117] "RemoveContainer" containerID="df2ab0487f972c944e31d2894ba376931efa0fd0d7ebe70f591119dd5d6d32d8"
Jul 16 23:06:33.763204 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:33.763170    2579 generic.go:356] "Generic (PLEG): container finished" podID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerID="9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed" exitCode=6
Jul 16 23:06:33.763613 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:33.763264    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerDied","Data":"9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed"}
Jul 16 23:06:33.763613 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:33.763316    2579 scope.go:117] "RemoveContainer" containerID="df2ab0487f972c944e31d2894ba376931efa0fd0d7ebe70f591119dd5d6d32d8"
Jul 16 23:06:33.763613 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:33.763401    2579 scope.go:117] "RemoveContainer" containerID="9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed"
Jul 16 23:06:33.763752 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:33.763611    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp_odh-ai-gateway-infra(32a9eaef-a168-4598-b418-c7bceeb99eb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6"
Jul 16 23:06:34.769484 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:34.769459    2579 scope.go:117] "RemoveContainer" containerID="9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed"
Jul 16 23:06:34.769869 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:34.769658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp_odh-ai-gateway-infra(32a9eaef-a168-4598-b418-c7bceeb99eb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6"
Jul 16 23:06:35.456892 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:35.456858    2579 scope.go:117] "RemoveContainer" containerID="9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4"
Jul 16 23:06:35.775569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:35.775490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerStarted","Data":"7fa3d831ad700a65ce10e1845a2a8daa7f0121925d3dee8ee9d23afbe8bfed1c"}
Jul 16 23:06:35.791443 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:35.791395    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" podStartSLOduration=18.791379271 podStartE2EDuration="18.791379271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:35.789534552 +0000 UTC m=+2957.857857605" watchObservedRunningTime="2026-07-16 23:06:35.791379271 +0000 UTC m=+2957.859702324"
Jul 16 23:06:36.781549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:36.781516    2579 generic.go:356] "Generic (PLEG): container finished" podID="ebcad4af-ef69-40da-9268-288f5bc83744" containerID="7fa3d831ad700a65ce10e1845a2a8daa7f0121925d3dee8ee9d23afbe8bfed1c" exitCode=6
Jul 16 23:06:36.782035 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:36.781589    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerDied","Data":"7fa3d831ad700a65ce10e1845a2a8daa7f0121925d3dee8ee9d23afbe8bfed1c"}
Jul 16 23:06:36.782035 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:36.781662    2579 scope.go:117] "RemoveContainer" containerID="9e35e0177a1e905497c3fa4b261d9a681a13e7e373693d5e15903e517bf43db4"
Jul 16 23:06:36.782035 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:36.782016    2579 scope.go:117] "RemoveContainer" containerID="7fa3d831ad700a65ce10e1845a2a8daa7f0121925d3dee8ee9d23afbe8bfed1c"
Jul 16 23:06:36.782215 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:36.782193    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq_odh-ai-gateway-infra(ebcad4af-ef69-40da-9268-288f5bc83744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" podUID="ebcad4af-ef69-40da-9268-288f5bc83744"
Jul 16 23:06:36.805580 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:36.805554    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:37.920018 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:37.919993    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:37.976184 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:37.976155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca\") pod \"ebcad4af-ef69-40da-9268-288f5bc83744\" (UID: \"ebcad4af-ef69-40da-9268-288f5bc83744\") "
Jul 16 23:06:37.976521 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:37.976500    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebcad4af-ef69-40da-9268-288f5bc83744" (UID: "ebcad4af-ef69-40da-9268-288f5bc83744"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:38.077085 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.077057    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebcad4af-ef69-40da-9268-288f5bc83744-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:06:38.793078 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.792986    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"
Jul 16 23:06:38.793078 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.793011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq" event={"ID":"ebcad4af-ef69-40da-9268-288f5bc83744","Type":"ContainerDied","Data":"45160ad73c53dba1e88ae8a816c919d30b2cf99d80eb39baa8e5ef662089a52a"}
Jul 16 23:06:38.793078 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.793072    2579 scope.go:117] "RemoveContainer" containerID="7fa3d831ad700a65ce10e1845a2a8daa7f0121925d3dee8ee9d23afbe8bfed1c"
Jul 16 23:06:38.810781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.810748    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:38.816281 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:38.816252    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x75gq"]
Jul 16 23:06:40.461746 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:40.461647    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" path="/var/lib/kubelet/pods/ebcad4af-ef69-40da-9268-288f5bc83744/volumes"
Jul 16 23:06:41.366174 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366139    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:06:41.366524 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366512    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366526    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366538    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366544    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366550    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366555    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366565    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366580    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366585    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366637    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366645    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366650    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366656    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.366782 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.366661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebcad4af-ef69-40da-9268-288f5bc83744" containerName="revoke-keys"
Jul 16 23:06:41.369693 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.369676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.371957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.371936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b5738f\"/\"openshift-service-ca.crt\""
Jul 16 23:06:41.372826 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.372809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b5738f\"/\"kube-root-ca.crt\""
Jul 16 23:06:41.372878 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.372840    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b5738f\"/\"auth-test-model-b5738f-kserve-self-signed-certs\""
Jul 16 23:06:41.372934 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.372914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b5738f\"/\"default-dockercfg-kmsqm\""
Jul 16 23:06:41.380505 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.380480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:06:41.403431 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.403537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.403537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.403537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.403537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.403669 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.403560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpx8g\" (UniqueName: \"kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.504979 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.504936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.504979 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.504989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpx8g\" (UniqueName: \"kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.505749 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.505546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.507222 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.507203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.507478 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.507458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.512984 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.512961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpx8g\" (UniqueName: \"kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g\") pod \"auth-test-model-b5738f-kserve-758fc74dff-dczl4\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") " pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.679302 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.679225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:41.809667 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:41.809644    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:06:41.811937 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:06:41.811908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc0ca4c2_fb9b_4c3b_b53f_0655aa0b98ab.slice/crio-ead846177bc130da647449a25efc966fdc6dc09d563ae3797ac09c07a687835c WatchSource:0}: Error finding container ead846177bc130da647449a25efc966fdc6dc09d563ae3797ac09c07a687835c: Status 404 returned error can't find the container with id ead846177bc130da647449a25efc966fdc6dc09d563ae3797ac09c07a687835c
Jul 16 23:06:42.817125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:42.817090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" event={"ID":"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab","Type":"ContainerStarted","Data":"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"}
Jul 16 23:06:42.817496 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:42.817132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" event={"ID":"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab","Type":"ContainerStarted","Data":"ead846177bc130da647449a25efc966fdc6dc09d563ae3797ac09c07a687835c"}
Jul 16 23:06:42.837258 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:42.837215    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" podStartSLOduration=1.837199499 podStartE2EDuration="1.837199499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:42.835773404 +0000 UTC m=+2964.904096461" watchObservedRunningTime="2026-07-16 23:06:42.837199499 +0000 UTC m=+2964.905522552"
Jul 16 23:06:47.456857 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:47.456825    2579 scope.go:117] "RemoveContainer" containerID="9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed"
Jul 16 23:06:47.843824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:47.843791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerStarted","Data":"bdce6d3455d169ef506468675401d0d8fee1f9c57ed3ce01a1f5417b931819f5"}
Jul 16 23:06:47.862294 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:47.862250    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" podStartSLOduration=17.862237638 podStartE2EDuration="17.862237638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:06:47.859814593 +0000 UTC m=+2969.928137646" watchObservedRunningTime="2026-07-16 23:06:47.862237638 +0000 UTC m=+2969.930560709"
Jul 16 23:06:48.850052 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:48.850023    2579 generic.go:356] "Generic (PLEG): container finished" podID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerID="bdce6d3455d169ef506468675401d0d8fee1f9c57ed3ce01a1f5417b931819f5" exitCode=6
Jul 16 23:06:48.850481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:48.850088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerDied","Data":"bdce6d3455d169ef506468675401d0d8fee1f9c57ed3ce01a1f5417b931819f5"}
Jul 16 23:06:48.850481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:48.850144    2579 scope.go:117] "RemoveContainer" containerID="9f163d510de7e3e82fb70e57164abb0361015f3afed8a71e31c7559aa5b1d3ed"
Jul 16 23:06:48.850481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:48.850405    2579 scope.go:117] "RemoveContainer" containerID="bdce6d3455d169ef506468675401d0d8fee1f9c57ed3ce01a1f5417b931819f5"
Jul 16 23:06:48.850688 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:06:48.850665    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp_odh-ai-gateway-infra(32a9eaef-a168-4598-b418-c7bceeb99eb6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6"
Jul 16 23:06:48.878544 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:48.878517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:49.988192 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:49.988170    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:50.077866 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.077836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca\") pod \"32a9eaef-a168-4598-b418-c7bceeb99eb6\" (UID: \"32a9eaef-a168-4598-b418-c7bceeb99eb6\") "
Jul 16 23:06:50.078246 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.078224    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "32a9eaef-a168-4598-b418-c7bceeb99eb6" (UID: "32a9eaef-a168-4598-b418-c7bceeb99eb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:06:50.179120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.179040    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32a9eaef-a168-4598-b418-c7bceeb99eb6-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:06:50.862674 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.862638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp" event={"ID":"32a9eaef-a168-4598-b418-c7bceeb99eb6","Type":"ContainerDied","Data":"eaffa8f34bd9ac17bf0e49e6fe7c6edaf327b2f0b7a74fbe245cebaeeec6b464"}
Jul 16 23:06:50.862863 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.862680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"
Jul 16 23:06:50.862863 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.862755    2579 scope.go:117] "RemoveContainer" containerID="bdce6d3455d169ef506468675401d0d8fee1f9c57ed3ce01a1f5417b931819f5"
Jul 16 23:06:50.879878 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.879848    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:50.885764 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:50.885739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-bq8pp"]
Jul 16 23:06:51.679761 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:51.679730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:51.680190 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:51.679777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:51.692313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:51.692289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:51.880194 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:51.880161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:06:52.463348 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:06:52.463315    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" path="/var/lib/kubelet/pods/32a9eaef-a168-4598-b418-c7bceeb99eb6/volumes"
Jul 16 23:07:19.225332 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225658    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225669    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225678    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225683    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225694    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225699    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225731    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:07:19.225738 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225737    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:07:19.225997 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225791    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1e7f96-b16e-4ff7-b9cb-b9e2529d7e98" containerName="revoke-keys"
Jul 16 23:07:19.225997 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225801    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225997 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225809    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.225997 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.225814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="32a9eaef-a168-4598-b418-c7bceeb99eb6" containerName="revoke-keys"
Jul 16 23:07:19.229911 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.229895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:19.232180 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.232162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:07:19.236629 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.236603    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:19.311066 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.311038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-vqdm8\" (UID: \"5da55460-a79a-4c60-9b7f-47477a1f859c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:19.412171 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.412146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-vqdm8\" (UID: \"5da55460-a79a-4c60-9b7f-47477a1f859c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:19.412755 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.412735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-vqdm8\" (UID: \"5da55460-a79a-4c60-9b7f-47477a1f859c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:19.542265 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.542197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:19.674825 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:19.674804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:19.676897 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:07:19.676859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da55460_a79a_4c60_9b7f_47477a1f859c.slice/crio-265544b6fdcffdf1552b6e093f3c104c4cd5152ecd7d5f2d2d504131038c7f04 WatchSource:0}: Error finding container 265544b6fdcffdf1552b6e093f3c104c4cd5152ecd7d5f2d2d504131038c7f04: Status 404 returned error can't find the container with id 265544b6fdcffdf1552b6e093f3c104c4cd5152ecd7d5f2d2d504131038c7f04
Jul 16 23:07:20.004468 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:20.004438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerStarted","Data":"148c0b91a89aca90420cde77c8a5065124a315474f617f2e4c09465fca81b378"}
Jul 16 23:07:20.004605 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:20.004497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerStarted","Data":"265544b6fdcffdf1552b6e093f3c104c4cd5152ecd7d5f2d2d504131038c7f04"}
Jul 16 23:07:20.020620 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:20.020575    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" podStartSLOduration=1.020561071 podStartE2EDuration="1.020561071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:07:20.019165005 +0000 UTC m=+3002.087488059" watchObservedRunningTime="2026-07-16 23:07:20.020561071 +0000 UTC m=+3002.088884125"
Jul 16 23:07:21.010384 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:21.010346    2579 generic.go:356] "Generic (PLEG): container finished" podID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerID="148c0b91a89aca90420cde77c8a5065124a315474f617f2e4c09465fca81b378" exitCode=6
Jul 16 23:07:21.010862 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:21.010410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerDied","Data":"148c0b91a89aca90420cde77c8a5065124a315474f617f2e4c09465fca81b378"}
Jul 16 23:07:21.010862 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:21.010547    2579 scope.go:117] "RemoveContainer" containerID="148c0b91a89aca90420cde77c8a5065124a315474f617f2e4c09465fca81b378"
Jul 16 23:07:22.016373 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:22.016281    2579 generic.go:356] "Generic (PLEG): container finished" podID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerID="3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa" exitCode=6
Jul 16 23:07:22.016373 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:22.016365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerDied","Data":"3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa"}
Jul 16 23:07:22.016860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:22.016406    2579 scope.go:117] "RemoveContainer" containerID="148c0b91a89aca90420cde77c8a5065124a315474f617f2e4c09465fca81b378"
Jul 16 23:07:22.016860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:22.016550    2579 scope.go:117] "RemoveContainer" containerID="3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa"
Jul 16 23:07:22.016860 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:07:22.016768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-vqdm8_odh-ai-gateway-infra(5da55460-a79a-4c60-9b7f-47477a1f859c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c"
Jul 16 23:07:23.023003 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:23.022978    2579 scope.go:117] "RemoveContainer" containerID="3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa"
Jul 16 23:07:23.023360 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:07:23.023141    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-vqdm8_odh-ai-gateway-infra(5da55460-a79a-4c60-9b7f-47477a1f859c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c"
Jul 16 23:07:33.456145 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:33.456112    2579 scope.go:117] "RemoveContainer" containerID="3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa"
Jul 16 23:07:34.077323 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:34.077296    2579 generic.go:356] "Generic (PLEG): container finished" podID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerID="d0b6d3993871737a3c310e11227c7a2620f5528fc04371fda818aebfa715d1bc" exitCode=6
Jul 16 23:07:34.077434 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:34.077373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerDied","Data":"d0b6d3993871737a3c310e11227c7a2620f5528fc04371fda818aebfa715d1bc"}
Jul 16 23:07:34.077477 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:34.077435    2579 scope.go:117] "RemoveContainer" containerID="3001ce9fa5ece57e30c19df96cbb2d46251a33b4431880cc76d0ccf82a5d7baa"
Jul 16 23:07:34.077768 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:34.077752    2579 scope.go:117] "RemoveContainer" containerID="d0b6d3993871737a3c310e11227c7a2620f5528fc04371fda818aebfa715d1bc"
Jul 16 23:07:34.077967 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:07:34.077945    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-vqdm8_odh-ai-gateway-infra(5da55460-a79a-4c60-9b7f-47477a1f859c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c"
Jul 16 23:07:35.103777 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:35.103744    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:35.240807 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:35.240783    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:35.335379 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:35.335354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca\") pod \"5da55460-a79a-4c60-9b7f-47477a1f859c\" (UID: \"5da55460-a79a-4c60-9b7f-47477a1f859c\") "
Jul 16 23:07:35.335672 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:35.335650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5da55460-a79a-4c60-9b7f-47477a1f859c" (UID: "5da55460-a79a-4c60-9b7f-47477a1f859c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:07:35.436585 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:35.436523    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5da55460-a79a-4c60-9b7f-47477a1f859c-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:07:36.088933 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.088908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"
Jul 16 23:07:36.089104 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.088906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8" event={"ID":"5da55460-a79a-4c60-9b7f-47477a1f859c","Type":"ContainerDied","Data":"265544b6fdcffdf1552b6e093f3c104c4cd5152ecd7d5f2d2d504131038c7f04"}
Jul 16 23:07:36.089104 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.089035    2579 scope.go:117] "RemoveContainer" containerID="d0b6d3993871737a3c310e11227c7a2620f5528fc04371fda818aebfa715d1bc"
Jul 16 23:07:36.109299 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.109273    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:36.114946 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.114925    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-vqdm8"]
Jul 16 23:07:36.461870 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:36.461800    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" path="/var/lib/kubelet/pods/5da55460-a79a-4c60-9b7f-47477a1f859c/volumes"
Jul 16 23:07:50.312701 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.312670    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:07:50.313111 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313083    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.313111 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313096    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.313184 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313118    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.313184 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.313246 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313206    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.313246 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.313216    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:07:50.317021 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.317004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:07:50.319306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.319286    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dspk5\""
Jul 16 23:07:50.322898 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.322870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:07:50.347635 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.347611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652n5\" (UniqueName: \"kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5\") pod \"kuadrant-operator-catalog-tldtt\" (UID: \"cf3f2d24-6887-4227-952f-6fa1b4a13d18\") " pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:07:50.448046 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.448022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652n5\" (UniqueName: \"kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5\") pod \"kuadrant-operator-catalog-tldtt\" (UID: \"cf3f2d24-6887-4227-952f-6fa1b4a13d18\") " pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:07:50.456769 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.456746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-652n5\" (UniqueName: \"kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5\") pod \"kuadrant-operator-catalog-tldtt\" (UID: \"cf3f2d24-6887-4227-952f-6fa1b4a13d18\") " pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:07:50.627553 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.627443    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:07:50.753057 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.753030    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:07:50.755780 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:07:50.755742    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3f2d24_6887_4227_952f_6fa1b4a13d18.slice/crio-740e94dfe794a0c49b23caece063c98aa33f108f66992d772b05b14b66f6233f WatchSource:0}: Error finding container 740e94dfe794a0c49b23caece063c98aa33f108f66992d772b05b14b66f6233f: Status 404 returned error can't find the container with id 740e94dfe794a0c49b23caece063c98aa33f108f66992d772b05b14b66f6233f
Jul 16 23:07:50.757290 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:50.757270    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 23:07:51.167408 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:51.167342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" event={"ID":"cf3f2d24-6887-4227-952f-6fa1b4a13d18","Type":"ContainerStarted","Data":"740e94dfe794a0c49b23caece063c98aa33f108f66992d772b05b14b66f6233f"}
Jul 16 23:07:53.179494 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:53.179463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" event={"ID":"cf3f2d24-6887-4227-952f-6fa1b4a13d18","Type":"ContainerStarted","Data":"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"}
Jul 16 23:07:53.194577 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:07:53.194527    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" podStartSLOduration=1.202589714 podStartE2EDuration="3.194513372s" totalImagesPullingTime="1.991923658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:07:53.193625923 +0000 UTC m=+3035.261948977" watchObservedRunningTime="2026-07-16 23:07:53.194513372 +0000 UTC m=+3035.262836427"
Jul 16 23:08:00.628208 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:00.628165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:00.628208 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:00.628223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:00.651242 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:00.651211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:01.238684 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:01.238659    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:01.271610 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:01.271584    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:08:03.224695 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:03.224654    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" podUID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" containerName="registry-server" containerID="cri-o://15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e" gracePeriod=2
Jul 16 23:08:03.471256 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:03.471236    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:03.560843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:03.560824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-652n5\" (UniqueName: \"kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5\") pod \"cf3f2d24-6887-4227-952f-6fa1b4a13d18\" (UID: \"cf3f2d24-6887-4227-952f-6fa1b4a13d18\") "
Jul 16 23:08:03.562663 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:03.562632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5" (OuterVolumeSpecName: "kube-api-access-652n5") pod "cf3f2d24-6887-4227-952f-6fa1b4a13d18" (UID: "cf3f2d24-6887-4227-952f-6fa1b4a13d18"). InnerVolumeSpecName "kube-api-access-652n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:08:03.662105 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:03.662085    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-652n5\" (UniqueName: \"kubernetes.io/projected/cf3f2d24-6887-4227-952f-6fa1b4a13d18-kube-api-access-652n5\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:08:04.230085 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.230055    2579 generic.go:356] "Generic (PLEG): container finished" podID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" containerID="15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e" exitCode=0
Jul 16 23:08:04.230461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.230094    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tldtt"
Jul 16 23:08:04.230461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.230141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" event={"ID":"cf3f2d24-6887-4227-952f-6fa1b4a13d18","Type":"ContainerDied","Data":"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"}
Jul 16 23:08:04.230461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.230189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tldtt" event={"ID":"cf3f2d24-6887-4227-952f-6fa1b4a13d18","Type":"ContainerDied","Data":"740e94dfe794a0c49b23caece063c98aa33f108f66992d772b05b14b66f6233f"}
Jul 16 23:08:04.230461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.230210    2579 scope.go:117] "RemoveContainer" containerID="15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"
Jul 16 23:08:04.239641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.239624    2579 scope.go:117] "RemoveContainer" containerID="15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"
Jul 16 23:08:04.239919 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:04.239901    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e\": container with ID starting with 15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e not found: ID does not exist" containerID="15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"
Jul 16 23:08:04.239986 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.239931    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e"} err="failed to get container status \"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e\": rpc error: code = NotFound desc = could not find container \"15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e\": container with ID starting with 15d0c1e5f81a35b5698f58b26590f09c5ca6ae01e0090f0ae0701659c404582e not found: ID does not exist"
Jul 16 23:08:04.250375 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.250355    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:08:04.253800 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.253780    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tldtt"]
Jul 16 23:08:04.462375 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:04.462348    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" path="/var/lib/kubelet/pods/cf3f2d24-6887-4227-952f-6fa1b4a13d18/volumes"
Jul 16 23:08:09.264759 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.264720    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:09.265253 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265231    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:08:09.265324 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:08:09.265324 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265283    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" containerName="registry-server"
Jul 16 23:08:09.265324 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265293    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" containerName="registry-server"
Jul 16 23:08:09.265536 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265393    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3f2d24-6887-4227-952f-6fa1b4a13d18" containerName="registry-server"
Jul 16 23:08:09.265536 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.265411    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da55460-a79a-4c60-9b7f-47477a1f859c" containerName="revoke-keys"
Jul 16 23:08:09.269871 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.269850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:09.272181 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.272163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:08:09.278537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.278503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:09.405228 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.405198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-tn9bw\" (UID: \"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:09.505943 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.505887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-tn9bw\" (UID: \"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:09.506439 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.506420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-tn9bw\" (UID: \"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:09.590684 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.590662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:09.717564 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:09.717539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:09.719551 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:08:09.719525    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac16cd4_eca1_43f5_ab96_ead5d0de8cc0.slice/crio-d1922008b02d43645796d03b3621d2e0aa00759e0014fc68bbc9b453c0e7ab37 WatchSource:0}: Error finding container d1922008b02d43645796d03b3621d2e0aa00759e0014fc68bbc9b453c0e7ab37: Status 404 returned error can't find the container with id d1922008b02d43645796d03b3621d2e0aa00759e0014fc68bbc9b453c0e7ab37
Jul 16 23:08:10.261572 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:10.261492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerStarted","Data":"12c1be4f6048b7fda6e799bf4fc0f94b2fa48d1efef949c1a2bb01a7a03e648b"}
Jul 16 23:08:10.261572 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:10.261549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerStarted","Data":"d1922008b02d43645796d03b3621d2e0aa00759e0014fc68bbc9b453c0e7ab37"}
Jul 16 23:08:10.278236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:10.278191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" podStartSLOduration=1.278178147 podStartE2EDuration="1.278178147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:08:10.276352253 +0000 UTC m=+3052.344675370" watchObservedRunningTime="2026-07-16 23:08:10.278178147 +0000 UTC m=+3052.346501211"
Jul 16 23:08:11.267193 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:11.267158    2579 generic.go:356] "Generic (PLEG): container finished" podID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerID="12c1be4f6048b7fda6e799bf4fc0f94b2fa48d1efef949c1a2bb01a7a03e648b" exitCode=6
Jul 16 23:08:11.267382 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:11.267240    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerDied","Data":"12c1be4f6048b7fda6e799bf4fc0f94b2fa48d1efef949c1a2bb01a7a03e648b"}
Jul 16 23:08:11.267382 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:11.267378    2579 scope.go:117] "RemoveContainer" containerID="12c1be4f6048b7fda6e799bf4fc0f94b2fa48d1efef949c1a2bb01a7a03e648b"
Jul 16 23:08:12.273774 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:12.273747    2579 generic.go:356] "Generic (PLEG): container finished" podID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerID="d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506" exitCode=6
Jul 16 23:08:12.274166 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:12.273827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerDied","Data":"d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506"}
Jul 16 23:08:12.274166 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:12.273892    2579 scope.go:117] "RemoveContainer" containerID="12c1be4f6048b7fda6e799bf4fc0f94b2fa48d1efef949c1a2bb01a7a03e648b"
Jul 16 23:08:12.274166 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:12.274046    2579 scope.go:117] "RemoveContainer" containerID="d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506"
Jul 16 23:08:12.274315 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:12.274234    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-tn9bw_odh-ai-gateway-infra(bac16cd4-eca1-43f5-ab96-ead5d0de8cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0"
Jul 16 23:08:13.280175 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:13.280148    2579 scope.go:117] "RemoveContainer" containerID="d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506"
Jul 16 23:08:13.280551 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:13.280345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-tn9bw_odh-ai-gateway-infra(bac16cd4-eca1-43f5-ab96-ead5d0de8cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0"
Jul 16 23:08:27.456883 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:27.456853    2579 scope.go:117] "RemoveContainer" containerID="d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506"
Jul 16 23:08:28.351656 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:28.351629    2579 generic.go:356] "Generic (PLEG): container finished" podID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerID="a0ff9984613bb8078118f74b913cf73f05f5284a8761a6bcf5906802aa232260" exitCode=6
Jul 16 23:08:28.351813 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:28.351702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerDied","Data":"a0ff9984613bb8078118f74b913cf73f05f5284a8761a6bcf5906802aa232260"}
Jul 16 23:08:28.351813 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:28.351783    2579 scope.go:117] "RemoveContainer" containerID="d2e4d51f589801879969752b7e4b05cede3f08d86c450ef643d99827c7207506"
Jul 16 23:08:28.352060 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:28.352037    2579 scope.go:117] "RemoveContainer" containerID="a0ff9984613bb8078118f74b913cf73f05f5284a8761a6bcf5906802aa232260"
Jul 16 23:08:28.352250 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:28.352231    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-tn9bw_odh-ai-gateway-infra(bac16cd4-eca1-43f5-ab96-ead5d0de8cc0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0"
Jul 16 23:08:29.383784 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:29.383749    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:29.515074 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:29.515053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:29.662643 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:29.662572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca\") pod \"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0\" (UID: \"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0\") "
Jul 16 23:08:29.662907 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:29.662886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" (UID: "bac16cd4-eca1-43f5-ab96-ead5d0de8cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:29.763958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:29.763933    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:08:30.364633 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.364600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw" event={"ID":"bac16cd4-eca1-43f5-ab96-ead5d0de8cc0","Type":"ContainerDied","Data":"d1922008b02d43645796d03b3621d2e0aa00759e0014fc68bbc9b453c0e7ab37"}
Jul 16 23:08:30.364633 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.364619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"
Jul 16 23:08:30.364887 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.364661    2579 scope.go:117] "RemoveContainer" containerID="a0ff9984613bb8078118f74b913cf73f05f5284a8761a6bcf5906802aa232260"
Jul 16 23:08:30.390972 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.390951    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:30.398274 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.398254    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-tn9bw"]
Jul 16 23:08:30.461974 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.461953    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" path="/var/lib/kubelet/pods/bac16cd4-eca1-43f5-ab96-ead5d0de8cc0/volumes"
Jul 16 23:08:30.947514 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947487    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:30.947857 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947844    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.947910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947859    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.947910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947871    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.947910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947877    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.947910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947884    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.947910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947889    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.948084 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947945    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.948084 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.947954    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:08:30.952066 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.952050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:30.954272 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.954257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:08:30.957824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:30.957802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:31.072150 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:31.072126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4\" (UID: \"26aa6a0e-092f-4f60-a4da-309b555af02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:31.180054 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:31.180018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4\" (UID: \"26aa6a0e-092f-4f60-a4da-309b555af02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:31.181120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:31.181100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4\" (UID: \"26aa6a0e-092f-4f60-a4da-309b555af02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:31.265007 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:31.264933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:31.394537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:31.394506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:31.395592 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:08:31.395567    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26aa6a0e_092f_4f60_a4da_309b555af02c.slice/crio-d49ebd27460820109b9ed34985a349c8400c2b8a1639f2d3d8737c93c702388a WatchSource:0}: Error finding container d49ebd27460820109b9ed34985a349c8400c2b8a1639f2d3d8737c93c702388a: Status 404 returned error can't find the container with id d49ebd27460820109b9ed34985a349c8400c2b8a1639f2d3d8737c93c702388a
Jul 16 23:08:32.379924 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:32.379888    2579 generic.go:356] "Generic (PLEG): container finished" podID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerID="72259f11919379b5b3eec7988950b7d3eb4c157b68a14490eb78aa2e757a8777" exitCode=6
Jul 16 23:08:32.380083 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:32.379943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" event={"ID":"26aa6a0e-092f-4f60-a4da-309b555af02c","Type":"ContainerDied","Data":"72259f11919379b5b3eec7988950b7d3eb4c157b68a14490eb78aa2e757a8777"}
Jul 16 23:08:32.380083 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:32.379991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" event={"ID":"26aa6a0e-092f-4f60-a4da-309b555af02c","Type":"ContainerStarted","Data":"d49ebd27460820109b9ed34985a349c8400c2b8a1639f2d3d8737c93c702388a"}
Jul 16 23:08:32.380201 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:32.380184    2579 scope.go:117] "RemoveContainer" containerID="72259f11919379b5b3eec7988950b7d3eb4c157b68a14490eb78aa2e757a8777"
Jul 16 23:08:33.386573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:33.386543    2579 generic.go:356] "Generic (PLEG): container finished" podID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerID="20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283" exitCode=6
Jul 16 23:08:33.387037 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:33.386632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" event={"ID":"26aa6a0e-092f-4f60-a4da-309b555af02c","Type":"ContainerDied","Data":"20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283"}
Jul 16 23:08:33.387037 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:33.386697    2579 scope.go:117] "RemoveContainer" containerID="72259f11919379b5b3eec7988950b7d3eb4c157b68a14490eb78aa2e757a8777"
Jul 16 23:08:33.387037 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:33.386859    2579 scope.go:117] "RemoveContainer" containerID="20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283"
Jul 16 23:08:33.387203 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:33.387052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4_odh-ai-gateway-infra(26aa6a0e-092f-4f60-a4da-309b555af02c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c"
Jul 16 23:08:34.392491 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:34.392466    2579 scope.go:117] "RemoveContainer" containerID="20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283"
Jul 16 23:08:34.392889 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:34.392628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4_odh-ai-gateway-infra(26aa6a0e-092f-4f60-a4da-309b555af02c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c"
Jul 16 23:08:48.459753 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:48.459719    2579 scope.go:117] "RemoveContainer" containerID="20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283"
Jul 16 23:08:49.468569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:49.468498    2579 generic.go:356] "Generic (PLEG): container finished" podID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerID="d162cc00bbe27289510f9090f747b9e6a0759832f1c6af0f3e94a581242bcb45" exitCode=6
Jul 16 23:08:49.468966 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:49.468568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" event={"ID":"26aa6a0e-092f-4f60-a4da-309b555af02c","Type":"ContainerDied","Data":"d162cc00bbe27289510f9090f747b9e6a0759832f1c6af0f3e94a581242bcb45"}
Jul 16 23:08:49.468966 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:49.468617    2579 scope.go:117] "RemoveContainer" containerID="20967db554ee15faf6b671fdf3742af13e8dff7368242371e420d9a92ff49283"
Jul 16 23:08:49.468966 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:49.468832    2579 scope.go:117] "RemoveContainer" containerID="d162cc00bbe27289510f9090f747b9e6a0759832f1c6af0f3e94a581242bcb45"
Jul 16 23:08:49.469098 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:08:49.468976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4_odh-ai-gateway-infra(26aa6a0e-092f-4f60-a4da-309b555af02c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c"
Jul 16 23:08:50.495515 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:50.495478    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:50.627954 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:50.627932    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:50.729450 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:50.729428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca\") pod \"26aa6a0e-092f-4f60-a4da-309b555af02c\" (UID: \"26aa6a0e-092f-4f60-a4da-309b555af02c\") "
Jul 16 23:08:50.729749 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:50.729729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26aa6a0e-092f-4f60-a4da-309b555af02c" (UID: "26aa6a0e-092f-4f60-a4da-309b555af02c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:08:50.830702 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:50.830679    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26aa6a0e-092f-4f60-a4da-309b555af02c-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:08:51.479605 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:51.479575    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"
Jul 16 23:08:51.479605 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:51.479591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4" event={"ID":"26aa6a0e-092f-4f60-a4da-309b555af02c","Type":"ContainerDied","Data":"d49ebd27460820109b9ed34985a349c8400c2b8a1639f2d3d8737c93c702388a"}
Jul 16 23:08:51.479870 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:51.479649    2579 scope.go:117] "RemoveContainer" containerID="d162cc00bbe27289510f9090f747b9e6a0759832f1c6af0f3e94a581242bcb45"
Jul 16 23:08:51.500372 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:51.500349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:51.511432 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:51.506358    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-zzsz4"]
Jul 16 23:08:52.462138 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:08:52.462100    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" path="/var/lib/kubelet/pods/26aa6a0e-092f-4f60-a4da-309b555af02c/volumes"
Jul 16 23:09:07.416966 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.416933    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417296    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417308    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417322    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417400    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417405 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417407    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.417594 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417415    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac16cd4-eca1-43f5-ab96-ead5d0de8cc0" containerName="revoke-keys"
Jul 16 23:09:07.417594 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.417421    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:07.420438 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.420421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:07.422747 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.422724    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:09:07.426929 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.426906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:07.563496 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.563465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz\" (UID: \"ba32df69-6560-429f-bc95-a822440e2bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:07.664682 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.664650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz\" (UID: \"ba32df69-6560-429f-bc95-a822440e2bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:07.665242 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.665224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz\" (UID: \"ba32df69-6560-429f-bc95-a822440e2bfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:07.732257 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.732204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:07.856012 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:07.855825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:07.858547 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:09:07.858508    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba32df69_6560_429f_bc95_a822440e2bfa.slice/crio-0ca01a7855de5ba0b7884ccdfada401dec9c8992bf65d0d6fa51cd89d1f3b847 WatchSource:0}: Error finding container 0ca01a7855de5ba0b7884ccdfada401dec9c8992bf65d0d6fa51cd89d1f3b847: Status 404 returned error can't find the container with id 0ca01a7855de5ba0b7884ccdfada401dec9c8992bf65d0d6fa51cd89d1f3b847
Jul 16 23:09:08.560502 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:08.560475    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba32df69-6560-429f-bc95-a822440e2bfa" containerID="d1c12ef11373aad0a70cc945cc4d722b0db90f9bcaca11579e9bf7d867a0e058" exitCode=6
Jul 16 23:09:08.560849 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:08.560560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" event={"ID":"ba32df69-6560-429f-bc95-a822440e2bfa","Type":"ContainerDied","Data":"d1c12ef11373aad0a70cc945cc4d722b0db90f9bcaca11579e9bf7d867a0e058"}
Jul 16 23:09:08.560849 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:08.560620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" event={"ID":"ba32df69-6560-429f-bc95-a822440e2bfa","Type":"ContainerStarted","Data":"0ca01a7855de5ba0b7884ccdfada401dec9c8992bf65d0d6fa51cd89d1f3b847"}
Jul 16 23:09:08.560849 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:08.560727    2579 scope.go:117] "RemoveContainer" containerID="d1c12ef11373aad0a70cc945cc4d722b0db90f9bcaca11579e9bf7d867a0e058"
Jul 16 23:09:09.567356 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:09.567324    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba32df69-6560-429f-bc95-a822440e2bfa" containerID="e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21" exitCode=6
Jul 16 23:09:09.567816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:09.567410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" event={"ID":"ba32df69-6560-429f-bc95-a822440e2bfa","Type":"ContainerDied","Data":"e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21"}
Jul 16 23:09:09.567816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:09.567477    2579 scope.go:117] "RemoveContainer" containerID="d1c12ef11373aad0a70cc945cc4d722b0db90f9bcaca11579e9bf7d867a0e058"
Jul 16 23:09:09.567816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:09.567548    2579 scope.go:117] "RemoveContainer" containerID="e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21"
Jul 16 23:09:09.567816 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:09.567770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz_odh-ai-gateway-infra(ba32df69-6560-429f-bc95-a822440e2bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" podUID="ba32df69-6560-429f-bc95-a822440e2bfa"
Jul 16 23:09:10.573201 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:10.573172    2579 scope.go:117] "RemoveContainer" containerID="e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21"
Jul 16 23:09:10.573575 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:10.573345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz_odh-ai-gateway-infra(ba32df69-6560-429f-bc95-a822440e2bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" podUID="ba32df69-6560-429f-bc95-a822440e2bfa"
Jul 16 23:09:25.456458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:25.456425    2579 scope.go:117] "RemoveContainer" containerID="e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21"
Jul 16 23:09:25.690435 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:25.690403    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:09:26.648601 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:26.648570    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba32df69-6560-429f-bc95-a822440e2bfa" containerID="07023c1ac910caaaf3ea9a2366d929beea6599e2b2568ba0c1901a39aae194d8" exitCode=6
Jul 16 23:09:26.649005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:26.648639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" event={"ID":"ba32df69-6560-429f-bc95-a822440e2bfa","Type":"ContainerDied","Data":"07023c1ac910caaaf3ea9a2366d929beea6599e2b2568ba0c1901a39aae194d8"}
Jul 16 23:09:26.649005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:26.648699    2579 scope.go:117] "RemoveContainer" containerID="e1d86443da989003bf3c65d12661d6ff80b0ed85d2ee07020f557c29bb8b8d21"
Jul 16 23:09:26.649005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:26.648968    2579 scope.go:117] "RemoveContainer" containerID="07023c1ac910caaaf3ea9a2366d929beea6599e2b2568ba0c1901a39aae194d8"
Jul 16 23:09:26.649201 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:26.649184    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz_odh-ai-gateway-infra(ba32df69-6560-429f-bc95-a822440e2bfa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" podUID="ba32df69-6560-429f-bc95-a822440e2bfa"
Jul 16 23:09:27.675236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:27.675201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:27.808231 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:27.808203    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:27.918202 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:27.918170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca\") pod \"ba32df69-6560-429f-bc95-a822440e2bfa\" (UID: \"ba32df69-6560-429f-bc95-a822440e2bfa\") "
Jul 16 23:09:27.918481 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:27.918460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba32df69-6560-429f-bc95-a822440e2bfa" (UID: "ba32df69-6560-429f-bc95-a822440e2bfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:09:28.018872 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.018817    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba32df69-6560-429f-bc95-a822440e2bfa-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:28.660358 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.660334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"
Jul 16 23:09:28.660358 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.660339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz" event={"ID":"ba32df69-6560-429f-bc95-a822440e2bfa","Type":"ContainerDied","Data":"0ca01a7855de5ba0b7884ccdfada401dec9c8992bf65d0d6fa51cd89d1f3b847"}
Jul 16 23:09:28.660530 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.660393    2579 scope.go:117] "RemoveContainer" containerID="07023c1ac910caaaf3ea9a2366d929beea6599e2b2568ba0c1901a39aae194d8"
Jul 16 23:09:28.676343 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.676317    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:28.680520 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:28.680496    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-s98pz"]
Jul 16 23:09:30.461728 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:30.461688    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" path="/var/lib/kubelet/pods/ba32df69-6560-429f-bc95-a822440e2bfa/volumes"
Jul 16 23:09:31.000289 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:31.000605 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000593    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.000650 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000607    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.000650 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.000650 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.000650 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000637    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:31.000650 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000643    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26aa6a0e-092f-4f60-a4da-309b555af02c" containerName="revoke-keys"
Jul 16 23:09:31.000831 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.000831 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.000742    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:31.004843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.004828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:31.007005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.006978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:09:31.009984 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.009940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:31.143541 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.143512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-774gn\" (UID: \"05e1998c-d49e-4c1f-9aff-d7adb14288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:31.244878 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.244848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-774gn\" (UID: \"05e1998c-d49e-4c1f-9aff-d7adb14288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:31.245392 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.245373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-774gn\" (UID: \"05e1998c-d49e-4c1f-9aff-d7adb14288f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:31.317573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.317539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:31.447463 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.447442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:31.449048 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:09:31.449020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e1998c_d49e_4c1f_9aff_d7adb14288f3.slice/crio-b7d9d32ba2660473f923a8097a5d6eccc89bbdb6a6a8009dd2c25c8be483144d WatchSource:0}: Error finding container b7d9d32ba2660473f923a8097a5d6eccc89bbdb6a6a8009dd2c25c8be483144d: Status 404 returned error can't find the container with id b7d9d32ba2660473f923a8097a5d6eccc89bbdb6a6a8009dd2c25c8be483144d
Jul 16 23:09:31.675906 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.675881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerStarted","Data":"b50072359aa1a889b88e3a02093c63fd8b29b1fe71b29ae7e4c9ba801eddcf74"}
Jul 16 23:09:31.676192 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.675927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerStarted","Data":"b7d9d32ba2660473f923a8097a5d6eccc89bbdb6a6a8009dd2c25c8be483144d"}
Jul 16 23:09:31.691774 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:31.691736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" podStartSLOduration=1.69172139 podStartE2EDuration="1.69172139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:31.689812752 +0000 UTC m=+3133.758135811" watchObservedRunningTime="2026-07-16 23:09:31.69172139 +0000 UTC m=+3133.760044435"
Jul 16 23:09:32.682159 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:32.682122    2579 generic.go:356] "Generic (PLEG): container finished" podID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerID="b50072359aa1a889b88e3a02093c63fd8b29b1fe71b29ae7e4c9ba801eddcf74" exitCode=6
Jul 16 23:09:32.682532 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:32.682208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerDied","Data":"b50072359aa1a889b88e3a02093c63fd8b29b1fe71b29ae7e4c9ba801eddcf74"}
Jul 16 23:09:32.682532 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:32.682337    2579 scope.go:117] "RemoveContainer" containerID="b50072359aa1a889b88e3a02093c63fd8b29b1fe71b29ae7e4c9ba801eddcf74"
Jul 16 23:09:33.689739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:33.689653    2579 generic.go:356] "Generic (PLEG): container finished" podID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerID="51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4" exitCode=6
Jul 16 23:09:33.689739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:33.689732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerDied","Data":"51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4"}
Jul 16 23:09:33.690128 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:33.689779    2579 scope.go:117] "RemoveContainer" containerID="b50072359aa1a889b88e3a02093c63fd8b29b1fe71b29ae7e4c9ba801eddcf74"
Jul 16 23:09:33.690128 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:33.689904    2579 scope.go:117] "RemoveContainer" containerID="51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4"
Jul 16 23:09:33.690128 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:33.690115    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-774gn_odh-ai-gateway-infra(05e1998c-d49e-4c1f-9aff-d7adb14288f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3"
Jul 16 23:09:34.695558 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:34.695530    2579 scope.go:117] "RemoveContainer" containerID="51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4"
Jul 16 23:09:34.695981 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:34.695685    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-774gn_odh-ai-gateway-infra(05e1998c-d49e-4c1f-9aff-d7adb14288f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3"
Jul 16 23:09:40.710215 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:40.710118    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" podUID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" containerName="main" containerID="cri-o://d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1" gracePeriod=45
Jul 16 23:09:40.958679 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:40.958644    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:09:41.125881 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.125854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.125894    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.125929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.125997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126028    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpx8g\" (UniqueName: \"kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir\") pod \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\" (UID: \"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab\") "
Jul 16 23:09:41.126511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home" (OuterVolumeSpecName: "home") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:09:41.126511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126255    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:09:41.126511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126379    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-model-cache\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.126511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126396    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-home\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.126511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.126397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:09:41.128276 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.128233    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:09:41.128276 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.128239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm" (OuterVolumeSpecName: "dshm") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:09:41.128474 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.128285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g" (OuterVolumeSpecName: "kube-api-access-mpx8g") pod "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" (UID: "bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab"). InnerVolumeSpecName "kube-api-access-mpx8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:09:41.227171 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.227138    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tls-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.227171 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.227166    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpx8g\" (UniqueName: \"kubernetes.io/projected/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-kube-api-access-mpx8g\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.227358 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.227179    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-tmp-dir\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.227358 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.227191    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab-dshm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:41.729622 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.729587    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" containerID="d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1" exitCode=0
Jul 16 23:09:41.730099 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.729631    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"
Jul 16 23:09:41.730099 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.729672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" event={"ID":"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab","Type":"ContainerDied","Data":"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"}
Jul 16 23:09:41.730099 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.729731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4" event={"ID":"bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab","Type":"ContainerDied","Data":"ead846177bc130da647449a25efc966fdc6dc09d563ae3797ac09c07a687835c"}
Jul 16 23:09:41.730099 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.729756    2579 scope.go:117] "RemoveContainer" containerID="d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"
Jul 16 23:09:41.738047 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.738025    2579 scope.go:117] "RemoveContainer" containerID="d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"
Jul 16 23:09:41.738298 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:41.738272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1\": container with ID starting with d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1 not found: ID does not exist" containerID="d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"
Jul 16 23:09:41.738346 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.738308    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1"} err="failed to get container status \"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1\": rpc error: code = NotFound desc = could not find container \"d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1\": container with ID starting with d4066eeaca2a970ae76355bbd13a5688ca8e6282e88ae864ced3321092ed52e1 not found: ID does not exist"
Jul 16 23:09:41.749643 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.749623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:09:41.753614 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:41.753593    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b5738f/auth-test-model-b5738f-kserve-758fc74dff-dczl4"]
Jul 16 23:09:42.461827 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:42.461797    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" path="/var/lib/kubelet/pods/bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab/volumes"
Jul 16 23:09:48.458638 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:48.458609    2579 scope.go:117] "RemoveContainer" containerID="51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4"
Jul 16 23:09:48.763494 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:48.763441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerStarted","Data":"01ed32ecfadc31f516346b699bfde246e988a4eda61f4806009039a0277076f2"}
Jul 16 23:09:49.769314 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:49.769275    2579 generic.go:356] "Generic (PLEG): container finished" podID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerID="01ed32ecfadc31f516346b699bfde246e988a4eda61f4806009039a0277076f2" exitCode=6
Jul 16 23:09:49.769730 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:49.769340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerDied","Data":"01ed32ecfadc31f516346b699bfde246e988a4eda61f4806009039a0277076f2"}
Jul 16 23:09:49.769730 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:49.769408    2579 scope.go:117] "RemoveContainer" containerID="51ae5857df29c20f9939f072f346d2be1b0f5100c37828845ba5bbb5f8983ca4"
Jul 16 23:09:49.769730 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:49.769636    2579 scope.go:117] "RemoveContainer" containerID="01ed32ecfadc31f516346b699bfde246e988a4eda61f4806009039a0277076f2"
Jul 16 23:09:49.769887 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:49.769848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-774gn_odh-ai-gateway-infra(05e1998c-d49e-4c1f-9aff-d7adb14288f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3"
Jul 16 23:09:49.793828 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:49.793807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:50.908247 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:50.908223    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:50.989006 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:50.988976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca\") pod \"05e1998c-d49e-4c1f-9aff-d7adb14288f3\" (UID: \"05e1998c-d49e-4c1f-9aff-d7adb14288f3\") "
Jul 16 23:09:50.989295 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:50.989271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05e1998c-d49e-4c1f-9aff-d7adb14288f3" (UID: "05e1998c-d49e-4c1f-9aff-d7adb14288f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:09:51.090430 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.090407    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e1998c-d49e-4c1f-9aff-d7adb14288f3-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:09:51.781122 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.781097    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"
Jul 16 23:09:51.781122 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.781103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn" event={"ID":"05e1998c-d49e-4c1f-9aff-d7adb14288f3","Type":"ContainerDied","Data":"b7d9d32ba2660473f923a8097a5d6eccc89bbdb6a6a8009dd2c25c8be483144d"}
Jul 16 23:09:51.781332 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.781157    2579 scope.go:117] "RemoveContainer" containerID="01ed32ecfadc31f516346b699bfde246e988a4eda61f4806009039a0277076f2"
Jul 16 23:09:51.803810 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.803790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:51.812514 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:51.812492    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-774gn"]
Jul 16 23:09:52.462185 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.462148    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" path="/var/lib/kubelet/pods/05e1998c-d49e-4c1f-9aff-d7adb14288f3/volumes"
Jul 16 23:09:52.774273 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774193    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:09:52.774616 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" containerName="main"
Jul 16 23:09:52.774696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774619    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" containerName="main"
Jul 16 23:09:52.774696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774632    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.774696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774640    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.774696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774652    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.774696 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774699    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774723    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774822    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774840    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba32df69-6560-429f-bc95-a822440e2bfa" containerName="revoke-keys"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774852    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc0ca4c2-fb9b-4c3b-b53f-0655aa0b98ab" containerName="main"
Jul 16 23:09:52.774958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.774865    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:52.779394 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.779373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:09:52.781912 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.781891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:09:52.784461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.784437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:09:52.804554 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.804531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz\" (UID: \"f2ee4222-a8b9-41d7-82f6-22c3425aa30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:09:52.905290 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.905266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz\" (UID: \"f2ee4222-a8b9-41d7-82f6-22c3425aa30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:09:52.905822 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:52.905805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz\" (UID: \"f2ee4222-a8b9-41d7-82f6-22c3425aa30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:09:53.068393 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.068366    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:09:53.068781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.068765    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:53.068865 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.068782    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:53.068922 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.068876    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e1998c-d49e-4c1f-9aff-d7adb14288f3" containerName="revoke-keys"
Jul 16 23:09:53.071970 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.071952    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.074990 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.074973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"openshift-service-ca.crt\""
Jul 16 23:09:53.075110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.074977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"default-dockercfg-rc8hh\""
Jul 16 23:09:53.075110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.075007    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"sub-test-model-61f04e-kserve-self-signed-certs\""
Jul 16 23:09:53.075110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.075038    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"kube-root-ca.crt\""
Jul 16 23:09:53.082876 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.082858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:09:53.092061 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.092043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:09:53.107799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.107772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.107909 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.107839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.107962 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.107923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.108013 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.107978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.108061 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.108008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwfk\" (UniqueName: \"kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.108061 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.108053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208718 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208870 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208870 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208985 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwfk\" (UniqueName: \"kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208985 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.208985 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.208973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.209192 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.209164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.209277 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.209239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.209573 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.209553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.211890 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.211863    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.212287 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.212232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.216845 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.216820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwfk\" (UniqueName: \"kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk\") pod \"sub-test-model-61f04e-kserve-59b87d4855-rjvqn\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") " pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.225367 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.225342    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:09:53.229229 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:09:53.229200    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ee4222_a8b9_41d7_82f6_22c3425aa30e.slice/crio-89f7f13f17f7e1a913b57b9be2262f0e79666a4353d5876d4e47cf3ebe75a464 WatchSource:0}: Error finding container 89f7f13f17f7e1a913b57b9be2262f0e79666a4353d5876d4e47cf3ebe75a464: Status 404 returned error can't find the container with id 89f7f13f17f7e1a913b57b9be2262f0e79666a4353d5876d4e47cf3ebe75a464
Jul 16 23:09:53.381921 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.381866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:09:53.515495 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.515470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:09:53.517371 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:09:53.517345    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3feeea56_7914_46c0_b7fe_662a73036c6c.slice/crio-2905892f66c7c5939395bbe9b77d2256e3dfaff1d68329cbf92cb13f4c621270 WatchSource:0}: Error finding container 2905892f66c7c5939395bbe9b77d2256e3dfaff1d68329cbf92cb13f4c621270: Status 404 returned error can't find the container with id 2905892f66c7c5939395bbe9b77d2256e3dfaff1d68329cbf92cb13f4c621270
Jul 16 23:09:53.794193 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.794097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" event={"ID":"3feeea56-7914-46c0-b7fe-662a73036c6c","Type":"ContainerStarted","Data":"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"}
Jul 16 23:09:53.794193 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.794140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" event={"ID":"3feeea56-7914-46c0-b7fe-662a73036c6c","Type":"ContainerStarted","Data":"2905892f66c7c5939395bbe9b77d2256e3dfaff1d68329cbf92cb13f4c621270"}
Jul 16 23:09:53.795483 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.795456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerStarted","Data":"292b579c9473d705ee92166f4ee6605a9e9ea6d196e6bc710e68b4370758a4e2"}
Jul 16 23:09:53.795595 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.795503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerStarted","Data":"89f7f13f17f7e1a913b57b9be2262f0e79666a4353d5876d4e47cf3ebe75a464"}
Jul 16 23:09:53.812008 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.811960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" podStartSLOduration=0.811945595 podStartE2EDuration="811.945595ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:53.810285231 +0000 UTC m=+3155.878608285" watchObservedRunningTime="2026-07-16 23:09:53.811945595 +0000 UTC m=+3155.880268648"
Jul 16 23:09:53.826900 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:53.826860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" podStartSLOduration=1.826847296 podStartE2EDuration="1.826847296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:09:53.825354967 +0000 UTC m=+3155.893678022" watchObservedRunningTime="2026-07-16 23:09:53.826847296 +0000 UTC m=+3155.895170373"
Jul 16 23:09:54.801691 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:54.801647    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerID="292b579c9473d705ee92166f4ee6605a9e9ea6d196e6bc710e68b4370758a4e2" exitCode=6
Jul 16 23:09:54.802155 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:54.801737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerDied","Data":"292b579c9473d705ee92166f4ee6605a9e9ea6d196e6bc710e68b4370758a4e2"}
Jul 16 23:09:54.802155 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:54.801968    2579 scope.go:117] "RemoveContainer" containerID="292b579c9473d705ee92166f4ee6605a9e9ea6d196e6bc710e68b4370758a4e2"
Jul 16 23:09:55.807687 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:55.807611    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerID="803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5" exitCode=6
Jul 16 23:09:55.807687 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:55.807683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerDied","Data":"803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5"}
Jul 16 23:09:55.808110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:55.807774    2579 scope.go:117] "RemoveContainer" containerID="292b579c9473d705ee92166f4ee6605a9e9ea6d196e6bc710e68b4370758a4e2"
Jul 16 23:09:55.808110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:55.807961    2579 scope.go:117] "RemoveContainer" containerID="803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5"
Jul 16 23:09:55.808191 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:55.808156    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz_odh-ai-gateway-infra(f2ee4222-a8b9-41d7-82f6-22c3425aa30e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e"
Jul 16 23:09:56.815636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:09:56.815609    2579 scope.go:117] "RemoveContainer" containerID="803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5"
Jul 16 23:09:56.815997 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:09:56.815823    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz_odh-ai-gateway-infra(f2ee4222-a8b9-41d7-82f6-22c3425aa30e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e"
Jul 16 23:10:03.382502 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:03.382461    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:03.382502 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:03.382512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:03.394771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:03.394750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:03.859659 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:03.859625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:07.458417 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.458384    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:07.461772 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.461757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:07.468013 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.467993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:07.520906 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.520874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf\" (UID: \"076ac8e3-862f-4e24-ae9f-6faf48fa2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:07.622610 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.622580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf\" (UID: \"076ac8e3-862f-4e24-ae9f-6faf48fa2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:07.623199 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.623181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf\" (UID: \"076ac8e3-862f-4e24-ae9f-6faf48fa2b92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:07.773694 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.773621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:07.902977 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:07.902949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:07.904385 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:10:07.904359    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076ac8e3_862f_4e24_ae9f_6faf48fa2b92.slice/crio-f2ac91d153eb1119a4481e30bd3c71ae9cf1ec8769bc79e9a04e7eaf184e02eb WatchSource:0}: Error finding container f2ac91d153eb1119a4481e30bd3c71ae9cf1ec8769bc79e9a04e7eaf184e02eb: Status 404 returned error can't find the container with id f2ac91d153eb1119a4481e30bd3c71ae9cf1ec8769bc79e9a04e7eaf184e02eb
Jul 16 23:10:08.875018 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:08.874990    2579 generic.go:356] "Generic (PLEG): container finished" podID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerID="66864ed1c8b89ff9a1842a4093b1b543e85c0dde89ba97d8a7b0eef3d6406ccf" exitCode=6
Jul 16 23:10:08.875387 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:08.875078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerDied","Data":"66864ed1c8b89ff9a1842a4093b1b543e85c0dde89ba97d8a7b0eef3d6406ccf"}
Jul 16 23:10:08.875387 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:08.875130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerStarted","Data":"f2ac91d153eb1119a4481e30bd3c71ae9cf1ec8769bc79e9a04e7eaf184e02eb"}
Jul 16 23:10:08.875387 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:08.875234    2579 scope.go:117] "RemoveContainer" containerID="66864ed1c8b89ff9a1842a4093b1b543e85c0dde89ba97d8a7b0eef3d6406ccf"
Jul 16 23:10:09.457005 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.456987    2579 scope.go:117] "RemoveContainer" containerID="803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5"
Jul 16 23:10:09.881306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.881282    2579 generic.go:356] "Generic (PLEG): container finished" podID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerID="906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a" exitCode=6
Jul 16 23:10:09.881727 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.881372    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerDied","Data":"906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a"}
Jul 16 23:10:09.881727 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.881434    2579 scope.go:117] "RemoveContainer" containerID="66864ed1c8b89ff9a1842a4093b1b543e85c0dde89ba97d8a7b0eef3d6406ccf"
Jul 16 23:10:09.881727 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.881544    2579 scope.go:117] "RemoveContainer" containerID="906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a"
Jul 16 23:10:09.881902 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:09.881740    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf_odh-ai-gateway-infra(076ac8e3-862f-4e24-ae9f-6faf48fa2b92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92"
Jul 16 23:10:09.883322 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:09.883301    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerStarted","Data":"b8cc7298508a68a74f5fced55961b1d40ab7d0b193e4a4fa8d26ed5e9f60715e"}
Jul 16 23:10:10.889977 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.889948    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerID="b8cc7298508a68a74f5fced55961b1d40ab7d0b193e4a4fa8d26ed5e9f60715e" exitCode=6
Jul 16 23:10:10.890415 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.890021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerDied","Data":"b8cc7298508a68a74f5fced55961b1d40ab7d0b193e4a4fa8d26ed5e9f60715e"}
Jul 16 23:10:10.890415 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.890077    2579 scope.go:117] "RemoveContainer" containerID="803d74c25fe2ea1bfe29907c61116a33a61bb876ec8ac55226b2ff3eb9bc48e5"
Jul 16 23:10:10.890415 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.890334    2579 scope.go:117] "RemoveContainer" containerID="b8cc7298508a68a74f5fced55961b1d40ab7d0b193e4a4fa8d26ed5e9f60715e"
Jul 16 23:10:10.890577 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:10.890530    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz_odh-ai-gateway-infra(f2ee4222-a8b9-41d7-82f6-22c3425aa30e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e"
Jul 16 23:10:10.892034 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.892020    2579 scope.go:117] "RemoveContainer" containerID="906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a"
Jul 16 23:10:10.892161 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:10.892146    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf_odh-ai-gateway-infra(076ac8e3-862f-4e24-ae9f-6faf48fa2b92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92"
Jul 16 23:10:10.920603 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:10.920578    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:10:12.029436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.029414    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:10:12.159533 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.159471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca\") pod \"f2ee4222-a8b9-41d7-82f6-22c3425aa30e\" (UID: \"f2ee4222-a8b9-41d7-82f6-22c3425aa30e\") "
Jul 16 23:10:12.159818 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.159798    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2ee4222-a8b9-41d7-82f6-22c3425aa30e" (UID: "f2ee4222-a8b9-41d7-82f6-22c3425aa30e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:12.260634 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.260604    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4222-a8b9-41d7-82f6-22c3425aa30e-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:12.904967 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.904942    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"
Jul 16 23:10:12.905125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.904967    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz" event={"ID":"f2ee4222-a8b9-41d7-82f6-22c3425aa30e","Type":"ContainerDied","Data":"89f7f13f17f7e1a913b57b9be2262f0e79666a4353d5876d4e47cf3ebe75a464"}
Jul 16 23:10:12.905125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.905028    2579 scope.go:117] "RemoveContainer" containerID="b8cc7298508a68a74f5fced55961b1d40ab7d0b193e4a4fa8d26ed5e9f60715e"
Jul 16 23:10:12.920815 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.920781    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:10:12.926635 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:12.926616    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-sqbqz"]
Jul 16 23:10:14.462386 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:14.462351    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" path="/var/lib/kubelet/pods/f2ee4222-a8b9-41d7-82f6-22c3425aa30e/volumes"
Jul 16 23:10:23.456492 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:23.456463    2579 scope.go:117] "RemoveContainer" containerID="906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a"
Jul 16 23:10:23.959613 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:23.959584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerStarted","Data":"4afff88491be7e3e3709865e7148eb84d9162b73c85576350cbb31dcaba57482"}
Jul 16 23:10:23.978739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:23.978674    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" podStartSLOduration=16.978660727 podStartE2EDuration="16.978660727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:10:23.976773958 +0000 UTC m=+3186.045097011" watchObservedRunningTime="2026-07-16 23:10:23.978660727 +0000 UTC m=+3186.046983780"
Jul 16 23:10:24.966024 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:24.965992    2579 generic.go:356] "Generic (PLEG): container finished" podID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerID="4afff88491be7e3e3709865e7148eb84d9162b73c85576350cbb31dcaba57482" exitCode=6
Jul 16 23:10:24.966485 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:24.966043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerDied","Data":"4afff88491be7e3e3709865e7148eb84d9162b73c85576350cbb31dcaba57482"}
Jul 16 23:10:24.966485 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:24.966088    2579 scope.go:117] "RemoveContainer" containerID="906847d290ae3947f1a2fe27676d06fee436ecc940455cbd136f02754a33361a"
Jul 16 23:10:24.966485 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:24.966336    2579 scope.go:117] "RemoveContainer" containerID="4afff88491be7e3e3709865e7148eb84d9162b73c85576350cbb31dcaba57482"
Jul 16 23:10:24.966618 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:24.966527    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf_odh-ai-gateway-infra(076ac8e3-862f-4e24-ae9f-6faf48fa2b92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92"
Jul 16 23:10:24.992209 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:24.992183    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:26.100943 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.100917    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:26.168209 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.168177    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca\") pod \"076ac8e3-862f-4e24-ae9f-6faf48fa2b92\" (UID: \"076ac8e3-862f-4e24-ae9f-6faf48fa2b92\") "
Jul 16 23:10:26.168538 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.168517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "076ac8e3-862f-4e24-ae9f-6faf48fa2b92" (UID: "076ac8e3-862f-4e24-ae9f-6faf48fa2b92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:26.269235 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.269185    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076ac8e3-862f-4e24-ae9f-6faf48fa2b92-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:26.978377 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.978348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"
Jul 16 23:10:26.978377 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.978369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf" event={"ID":"076ac8e3-862f-4e24-ae9f-6faf48fa2b92","Type":"ContainerDied","Data":"f2ac91d153eb1119a4481e30bd3c71ae9cf1ec8769bc79e9a04e7eaf184e02eb"}
Jul 16 23:10:26.978616 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.978425    2579 scope.go:117] "RemoveContainer" containerID="4afff88491be7e3e3709865e7148eb84d9162b73c85576350cbb31dcaba57482"
Jul 16 23:10:26.994692 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:26.994671    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:27.001194 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:27.001172    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-x88nf"]
Jul 16 23:10:28.462203 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:28.462171    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" path="/var/lib/kubelet/pods/076ac8e3-862f-4e24-ae9f-6faf48fa2b92/volumes"
Jul 16 23:10:29.454751 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:29.454718    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:10:31.050375 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050342    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050664    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050683    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050689    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050696    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050701    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050729    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050735    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050746    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050760    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.050781 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050764    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.051189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.051189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.051189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050828    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:31.051189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050834    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.051189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.050839    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="076ac8e3-862f-4e24-ae9f-6faf48fa2b92" containerName="revoke-keys"
Jul 16 23:10:31.053772 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.053754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:31.055912 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.055890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:10:31.061340 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.061318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:31.208809 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.208782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l\" (UID: \"9f1f8d39-95c1-41ad-b0f9-779f86d9631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:31.309309 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.309234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l\" (UID: \"9f1f8d39-95c1-41ad-b0f9-779f86d9631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:31.309852 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.309834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l\" (UID: \"9f1f8d39-95c1-41ad-b0f9-779f86d9631b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:31.364787 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.364766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:31.696871 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:31.696847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:31.698986 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:10:31.698965    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1f8d39_95c1_41ad_b0f9_779f86d9631b.slice/crio-1b9f8658f42531b7480020a38ed2fc53eb427cf939a04f3e7cad401e34acd97c WatchSource:0}: Error finding container 1b9f8658f42531b7480020a38ed2fc53eb427cf939a04f3e7cad401e34acd97c: Status 404 returned error can't find the container with id 1b9f8658f42531b7480020a38ed2fc53eb427cf939a04f3e7cad401e34acd97c
Jul 16 23:10:32.003989 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:32.003932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerStarted","Data":"c6993031b4952baf4b59226a65a6a63272b95d357e5cf832e14fb36832dab502"}
Jul 16 23:10:32.003989 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:32.003978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerStarted","Data":"1b9f8658f42531b7480020a38ed2fc53eb427cf939a04f3e7cad401e34acd97c"}
Jul 16 23:10:32.020763 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:32.020720    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" podStartSLOduration=1.020693861 podStartE2EDuration="1.020693861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:10:32.019891232 +0000 UTC m=+3194.088214287" watchObservedRunningTime="2026-07-16 23:10:32.020693861 +0000 UTC m=+3194.089016915"
Jul 16 23:10:33.009619 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:33.009584    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerID="c6993031b4952baf4b59226a65a6a63272b95d357e5cf832e14fb36832dab502" exitCode=6
Jul 16 23:10:33.010088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:33.009661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerDied","Data":"c6993031b4952baf4b59226a65a6a63272b95d357e5cf832e14fb36832dab502"}
Jul 16 23:10:33.010088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:33.009847    2579 scope.go:117] "RemoveContainer" containerID="c6993031b4952baf4b59226a65a6a63272b95d357e5cf832e14fb36832dab502"
Jul 16 23:10:34.016526 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:34.016453    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerID="c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847" exitCode=6
Jul 16 23:10:34.016526 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:34.016514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerDied","Data":"c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847"}
Jul 16 23:10:34.017033 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:34.016574    2579 scope.go:117] "RemoveContainer" containerID="c6993031b4952baf4b59226a65a6a63272b95d357e5cf832e14fb36832dab502"
Jul 16 23:10:34.017033 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:34.016768    2579 scope.go:117] "RemoveContainer" containerID="c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847"
Jul 16 23:10:34.017033 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:34.016985    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l_odh-ai-gateway-infra(9f1f8d39-95c1-41ad-b0f9-779f86d9631b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b"
Jul 16 23:10:35.022922 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:35.022889    2579 scope.go:117] "RemoveContainer" containerID="c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847"
Jul 16 23:10:35.023379 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:35.023136    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l_odh-ai-gateway-infra(9f1f8d39-95c1-41ad-b0f9-779f86d9631b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b"
Jul 16 23:10:44.476659 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.476625    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" podUID="3feeea56-7914-46c0-b7fe-662a73036c6c" containerName="main" containerID="cri-o://7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc" gracePeriod=45
Jul 16 23:10:44.730817 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.730771    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:44.910614 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.910788 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.910788 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpwfk\" (UniqueName: \"kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.910788 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.910969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.910969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:10:44.910969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:10:44.910969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.910943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home\") pod \"3feeea56-7914-46c0-b7fe-662a73036c6c\" (UID: \"3feeea56-7914-46c0-b7fe-662a73036c6c\") "
Jul 16 23:10:44.911253 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.911180    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-model-cache\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:44.911253 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.911200    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-tmp-dir\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:44.911253 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.911192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home" (OuterVolumeSpecName: "home") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:10:44.913070 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.913044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:10:44.913165 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.913124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk" (OuterVolumeSpecName: "kube-api-access-dpwfk") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "kube-api-access-dpwfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:10:44.913165 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:44.913127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm" (OuterVolumeSpecName: "dshm") pod "3feeea56-7914-46c0-b7fe-662a73036c6c" (UID: "3feeea56-7914-46c0-b7fe-662a73036c6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:10:45.012329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.012273    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwfk\" (UniqueName: \"kubernetes.io/projected/3feeea56-7914-46c0-b7fe-662a73036c6c-kube-api-access-dpwfk\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:45.012329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.012297    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-dshm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:45.012329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.012308    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3feeea56-7914-46c0-b7fe-662a73036c6c-tls-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:45.012329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.012315    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3feeea56-7914-46c0-b7fe-662a73036c6c-home\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:45.070039 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.070015    2579 generic.go:356] "Generic (PLEG): container finished" podID="3feeea56-7914-46c0-b7fe-662a73036c6c" containerID="7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc" exitCode=0
Jul 16 23:10:45.070137 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.070056    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"
Jul 16 23:10:45.070137 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.070105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" event={"ID":"3feeea56-7914-46c0-b7fe-662a73036c6c","Type":"ContainerDied","Data":"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"}
Jul 16 23:10:45.070209 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.070142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn" event={"ID":"3feeea56-7914-46c0-b7fe-662a73036c6c","Type":"ContainerDied","Data":"2905892f66c7c5939395bbe9b77d2256e3dfaff1d68329cbf92cb13f4c621270"}
Jul 16 23:10:45.070209 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.070160    2579 scope.go:117] "RemoveContainer" containerID="7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"
Jul 16 23:10:45.078882 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.078866    2579 scope.go:117] "RemoveContainer" containerID="7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"
Jul 16 23:10:45.079141 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:45.079122    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc\": container with ID starting with 7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc not found: ID does not exist" containerID="7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"
Jul 16 23:10:45.079201 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.079147    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc"} err="failed to get container status \"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc\": rpc error: code = NotFound desc = could not find container \"7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc\": container with ID starting with 7ebf4fcbed42453f421935de32f7b557fcdbec3374a146afec5915aaf3f813bc not found: ID does not exist"
Jul 16 23:10:45.092242 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.092219    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:10:45.094769 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:45.094750    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/sub-test-model-61f04e-kserve-59b87d4855-rjvqn"]
Jul 16 23:10:46.462002 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:46.461970    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3feeea56-7914-46c0-b7fe-662a73036c6c" path="/var/lib/kubelet/pods/3feeea56-7914-46c0-b7fe-662a73036c6c/volumes"
Jul 16 23:10:50.456789 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:50.456756    2579 scope.go:117] "RemoveContainer" containerID="c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847"
Jul 16 23:10:51.101545 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:51.101513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerStarted","Data":"df757a8b67807982167f506d742d9eb896dd043837eef482098ae86e7a40b699"}
Jul 16 23:10:52.107660 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.107627    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerID="df757a8b67807982167f506d742d9eb896dd043837eef482098ae86e7a40b699" exitCode=6
Jul 16 23:10:52.108075 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.107700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerDied","Data":"df757a8b67807982167f506d742d9eb896dd043837eef482098ae86e7a40b699"}
Jul 16 23:10:52.108075 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.107781    2579 scope.go:117] "RemoveContainer" containerID="c9e39ec93b4d596921c38689ba74a3cb1e9efcbac1126c10ddeb42ca4f815847"
Jul 16 23:10:52.108174 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.108115    2579 scope.go:117] "RemoveContainer" containerID="df757a8b67807982167f506d742d9eb896dd043837eef482098ae86e7a40b699"
Jul 16 23:10:52.108326 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:52.108302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l_odh-ai-gateway-infra(9f1f8d39-95c1-41ad-b0f9-779f86d9631b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b"
Jul 16 23:10:52.130721 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.130684    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:52.816794 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.816757    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:10:52.817263 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.817245    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3feeea56-7914-46c0-b7fe-662a73036c6c" containerName="main"
Jul 16 23:10:52.817311 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.817267    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3feeea56-7914-46c0-b7fe-662a73036c6c" containerName="main"
Jul 16 23:10:52.817401 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.817389    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3feeea56-7914-46c0-b7fe-662a73036c6c" containerName="main"
Jul 16 23:10:52.817437 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.817407    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ee4222-a8b9-41d7-82f6-22c3425aa30e" containerName="revoke-keys"
Jul 16 23:10:52.822001 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.821981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:10:52.826516 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.826492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:10:52.874743 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.874694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w\" (UID: \"8aa1b63d-e03e-4087-8681-5d088f7a1838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:10:52.975791 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.975768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w\" (UID: \"8aa1b63d-e03e-4087-8681-5d088f7a1838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:10:52.976371 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:52.976353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w\" (UID: \"8aa1b63d-e03e-4087-8681-5d088f7a1838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:10:53.133275 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.133246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:10:53.259479 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.259456    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:53.278444 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.278418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca\") pod \"9f1f8d39-95c1-41ad-b0f9-779f86d9631b\" (UID: \"9f1f8d39-95c1-41ad-b0f9-779f86d9631b\") "
Jul 16 23:10:53.278882 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.278859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f1f8d39-95c1-41ad-b0f9-779f86d9631b" (UID: "9f1f8d39-95c1-41ad-b0f9-779f86d9631b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:10:53.380071 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.380008    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f1f8d39-95c1-41ad-b0f9-779f86d9631b-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:10:53.477048 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:53.476892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:10:53.479038 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:10:53.479007    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa1b63d_e03e_4087_8681_5d088f7a1838.slice/crio-ab2e102291b433e7181cfea2a0398cc144834205afaa6ea0d7a8f2f4801fb3d9 WatchSource:0}: Error finding container ab2e102291b433e7181cfea2a0398cc144834205afaa6ea0d7a8f2f4801fb3d9: Status 404 returned error can't find the container with id ab2e102291b433e7181cfea2a0398cc144834205afaa6ea0d7a8f2f4801fb3d9
Jul 16 23:10:54.118315 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.118293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"
Jul 16 23:10:54.118436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.118340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l" event={"ID":"9f1f8d39-95c1-41ad-b0f9-779f86d9631b","Type":"ContainerDied","Data":"1b9f8658f42531b7480020a38ed2fc53eb427cf939a04f3e7cad401e34acd97c"}
Jul 16 23:10:54.118436 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.118396    2579 scope.go:117] "RemoveContainer" containerID="df757a8b67807982167f506d742d9eb896dd043837eef482098ae86e7a40b699"
Jul 16 23:10:54.119957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.119935    2579 generic.go:356] "Generic (PLEG): container finished" podID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerID="7dc58ff7607b2f3210ed2333437cbb93fdb27eced9a52d8261c2fc96ebb48a69" exitCode=6
Jul 16 23:10:54.120058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.119972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" event={"ID":"8aa1b63d-e03e-4087-8681-5d088f7a1838","Type":"ContainerDied","Data":"7dc58ff7607b2f3210ed2333437cbb93fdb27eced9a52d8261c2fc96ebb48a69"}
Jul 16 23:10:54.120058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.120012    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" event={"ID":"8aa1b63d-e03e-4087-8681-5d088f7a1838","Type":"ContainerStarted","Data":"ab2e102291b433e7181cfea2a0398cc144834205afaa6ea0d7a8f2f4801fb3d9"}
Jul 16 23:10:54.120165 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.120152    2579 scope.go:117] "RemoveContainer" containerID="7dc58ff7607b2f3210ed2333437cbb93fdb27eced9a52d8261c2fc96ebb48a69"
Jul 16 23:10:54.150679 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.150659    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:54.152771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.152751    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-2lb7l"]
Jul 16 23:10:54.466889 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:54.466863    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" path="/var/lib/kubelet/pods/9f1f8d39-95c1-41ad-b0f9-779f86d9631b/volumes"
Jul 16 23:10:55.126499 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:55.126422    2579 generic.go:356] "Generic (PLEG): container finished" podID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerID="ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256" exitCode=6
Jul 16 23:10:55.126653 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:55.126513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" event={"ID":"8aa1b63d-e03e-4087-8681-5d088f7a1838","Type":"ContainerDied","Data":"ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256"}
Jul 16 23:10:55.126653 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:55.126585    2579 scope.go:117] "RemoveContainer" containerID="7dc58ff7607b2f3210ed2333437cbb93fdb27eced9a52d8261c2fc96ebb48a69"
Jul 16 23:10:55.126810 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:55.126688    2579 scope.go:117] "RemoveContainer" containerID="ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256"
Jul 16 23:10:55.126933 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:55.126904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w_odh-ai-gateway-infra(8aa1b63d-e03e-4087-8681-5d088f7a1838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838"
Jul 16 23:10:56.133854 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:56.133830    2579 scope.go:117] "RemoveContainer" containerID="ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256"
Jul 16 23:10:56.134214 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:10:56.133992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w_odh-ai-gateway-infra(8aa1b63d-e03e-4087-8681-5d088f7a1838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838"
Jul 16 23:10:57.493604 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.493569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:10:57.493977 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.493947    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.493977 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.493962    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.493977 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.493975    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.494120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.493981    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.494120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.494043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.494120 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.494053    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:10:57.498156 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.498140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:10:57.503989 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.503963    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:10:57.511036 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.511008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d\" (UID: \"d3a91367-b462-4ea9-a7ff-2c55b7348416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:10:57.612484 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.612455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d\" (UID: \"d3a91367-b462-4ea9-a7ff-2c55b7348416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:10:57.613013 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.612996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d\" (UID: \"d3a91367-b462-4ea9-a7ff-2c55b7348416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:10:57.809858 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.809780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:10:57.936454 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:57.936332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:10:57.938520 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:10:57.938491    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a91367_b462_4ea9_a7ff_2c55b7348416.slice/crio-a3fd503240e10bea44952c03c90168e2ac104bb92a87ea2836be54d1a567fcad WatchSource:0}: Error finding container a3fd503240e10bea44952c03c90168e2ac104bb92a87ea2836be54d1a567fcad: Status 404 returned error can't find the container with id a3fd503240e10bea44952c03c90168e2ac104bb92a87ea2836be54d1a567fcad
Jul 16 23:10:58.145524 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:58.145497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerStarted","Data":"723f2cc003a8516a9b373c9c5f1f2d0fd3de912936bc4f069c412a4e09260ad2"}
Jul 16 23:10:58.145638 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:58.145546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerStarted","Data":"a3fd503240e10bea44952c03c90168e2ac104bb92a87ea2836be54d1a567fcad"}
Jul 16 23:10:58.160159 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:58.160120    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" podStartSLOduration=1.160107368 podStartE2EDuration="1.160107368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:10:58.158113171 +0000 UTC m=+3220.226436228" watchObservedRunningTime="2026-07-16 23:10:58.160107368 +0000 UTC m=+3220.228430422"
Jul 16 23:10:59.151996 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:59.151966    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerID="723f2cc003a8516a9b373c9c5f1f2d0fd3de912936bc4f069c412a4e09260ad2" exitCode=6
Jul 16 23:10:59.152431 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:59.152018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerDied","Data":"723f2cc003a8516a9b373c9c5f1f2d0fd3de912936bc4f069c412a4e09260ad2"}
Jul 16 23:10:59.152431 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:10:59.152221    2579 scope.go:117] "RemoveContainer" containerID="723f2cc003a8516a9b373c9c5f1f2d0fd3de912936bc4f069c412a4e09260ad2"
Jul 16 23:11:00.158307 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:00.158276    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerID="e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c" exitCode=6
Jul 16 23:11:00.158689 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:00.158362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerDied","Data":"e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c"}
Jul 16 23:11:00.158689 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:00.158423    2579 scope.go:117] "RemoveContainer" containerID="723f2cc003a8516a9b373c9c5f1f2d0fd3de912936bc4f069c412a4e09260ad2"
Jul 16 23:11:00.158689 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:00.158577    2579 scope.go:117] "RemoveContainer" containerID="e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c"
Jul 16 23:11:00.158847 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:00.158795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d_odh-ai-gateway-infra(d3a91367-b462-4ea9-a7ff-2c55b7348416)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416"
Jul 16 23:11:01.164899 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:01.164875    2579 scope.go:117] "RemoveContainer" containerID="e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c"
Jul 16 23:11:01.165348 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:01.165025    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d_odh-ai-gateway-infra(d3a91367-b462-4ea9-a7ff-2c55b7348416)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416"
Jul 16 23:11:09.407085 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.407052    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:09.407466 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.407401    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:11:09.407466 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.407411    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:11:09.407538 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.407481    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f1f8d39-95c1-41ad-b0f9-779f86d9631b" containerName="revoke-keys"
Jul 16 23:11:09.411765 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.411748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:09.417758 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.417694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:09.498831 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.498756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-6zsmx\" (UID: \"7b68c497-ef67-4fda-b0a9-e8488ee5b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:09.599932 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.599903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-6zsmx\" (UID: \"7b68c497-ef67-4fda-b0a9-e8488ee5b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:09.600483 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.600465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-6zsmx\" (UID: \"7b68c497-ef67-4fda-b0a9-e8488ee5b086\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:09.723552 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.723477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:09.849590 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:09.849563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:09.851266 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:11:09.851238    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b68c497_ef67_4fda_b0a9_e8488ee5b086.slice/crio-b533671949adc872563971506639c77635472b9917015f98ca89b7f5aae2661f WatchSource:0}: Error finding container b533671949adc872563971506639c77635472b9917015f98ca89b7f5aae2661f: Status 404 returned error can't find the container with id b533671949adc872563971506639c77635472b9917015f98ca89b7f5aae2661f
Jul 16 23:11:10.208511 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:10.208480    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerStarted","Data":"001258668e8f3f9e2d7e72d2adaf122818b6652b607a6c6ee6a0e6297f01c312"}
Jul 16 23:11:10.208612 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:10.208564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerStarted","Data":"b533671949adc872563971506639c77635472b9917015f98ca89b7f5aae2661f"}
Jul 16 23:11:10.225011 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:10.224959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" podStartSLOduration=1.224946187 podStartE2EDuration="1.224946187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:11:10.223342882 +0000 UTC m=+3232.291665938" watchObservedRunningTime="2026-07-16 23:11:10.224946187 +0000 UTC m=+3232.293269240"
Jul 16 23:11:10.456437 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:10.456413    2579 scope.go:117] "RemoveContainer" containerID="ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256"
Jul 16 23:11:11.216004 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.215979    2579 generic.go:356] "Generic (PLEG): container finished" podID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerID="3fb6b0dff23404813ce0c82dfeead1ea85a7475798f391f0eb96f7953efefb10" exitCode=6
Jul 16 23:11:11.216129 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.216051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" event={"ID":"8aa1b63d-e03e-4087-8681-5d088f7a1838","Type":"ContainerDied","Data":"3fb6b0dff23404813ce0c82dfeead1ea85a7475798f391f0eb96f7953efefb10"}
Jul 16 23:11:11.216189 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.216128    2579 scope.go:117] "RemoveContainer" containerID="ff179c75ed3df1cbae37f16c9d9dfc4e4ef71b8286ef87c2a527d158c0f43256"
Jul 16 23:11:11.216455 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.216420    2579 scope.go:117] "RemoveContainer" containerID="3fb6b0dff23404813ce0c82dfeead1ea85a7475798f391f0eb96f7953efefb10"
Jul 16 23:11:11.216626 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:11.216607    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w_odh-ai-gateway-infra(8aa1b63d-e03e-4087-8681-5d088f7a1838)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838"
Jul 16 23:11:11.217664 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.217645    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerID="001258668e8f3f9e2d7e72d2adaf122818b6652b607a6c6ee6a0e6297f01c312" exitCode=6
Jul 16 23:11:11.217778 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.217678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerDied","Data":"001258668e8f3f9e2d7e72d2adaf122818b6652b607a6c6ee6a0e6297f01c312"}
Jul 16 23:11:11.217893 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:11.217880    2579 scope.go:117] "RemoveContainer" containerID="001258668e8f3f9e2d7e72d2adaf122818b6652b607a6c6ee6a0e6297f01c312"
Jul 16 23:11:12.226586 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.226507    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerID="53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc" exitCode=6
Jul 16 23:11:12.227038 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.226593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerDied","Data":"53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc"}
Jul 16 23:11:12.227038 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.226661    2579 scope.go:117] "RemoveContainer" containerID="001258668e8f3f9e2d7e72d2adaf122818b6652b607a6c6ee6a0e6297f01c312"
Jul 16 23:11:12.227038 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.226845    2579 scope.go:117] "RemoveContainer" containerID="53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc"
Jul 16 23:11:12.227182 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:12.227078    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-6zsmx_odh-ai-gateway-infra(7b68c497-ef67-4fda-b0a9-e8488ee5b086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086"
Jul 16 23:11:12.242049 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.242028    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:11:12.370453 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.370432    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:11:12.422285 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.422260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca\") pod \"8aa1b63d-e03e-4087-8681-5d088f7a1838\" (UID: \"8aa1b63d-e03e-4087-8681-5d088f7a1838\") "
Jul 16 23:11:12.422608 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.422586    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8aa1b63d-e03e-4087-8681-5d088f7a1838" (UID: "8aa1b63d-e03e-4087-8681-5d088f7a1838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:12.523486 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:12.523434    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa1b63d-e03e-4087-8681-5d088f7a1838-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:11:13.232292 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.232254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"
Jul 16 23:11:13.232776 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.232256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w" event={"ID":"8aa1b63d-e03e-4087-8681-5d088f7a1838","Type":"ContainerDied","Data":"ab2e102291b433e7181cfea2a0398cc144834205afaa6ea0d7a8f2f4801fb3d9"}
Jul 16 23:11:13.232776 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.232410    2579 scope.go:117] "RemoveContainer" containerID="3fb6b0dff23404813ce0c82dfeead1ea85a7475798f391f0eb96f7953efefb10"
Jul 16 23:11:13.234358 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.234328    2579 scope.go:117] "RemoveContainer" containerID="53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc"
Jul 16 23:11:13.234564 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:13.234539    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-6zsmx_odh-ai-gateway-infra(7b68c497-ef67-4fda-b0a9-e8488ee5b086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086"
Jul 16 23:11:13.263054 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.263033    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:11:13.264743 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:13.264725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-crt6w"]
Jul 16 23:11:14.456489 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:14.456463    2579 scope.go:117] "RemoveContainer" containerID="e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c"
Jul 16 23:11:14.461687 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:14.461663    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" path="/var/lib/kubelet/pods/8aa1b63d-e03e-4087-8681-5d088f7a1838/volumes"
Jul 16 23:11:15.247141 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:15.247117    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerID="500af4646c3cd2f0758baed56fc49801d8d0d1a0ba19bf2c55907fdca80d4026" exitCode=6
Jul 16 23:11:15.247245 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:15.247196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerDied","Data":"500af4646c3cd2f0758baed56fc49801d8d0d1a0ba19bf2c55907fdca80d4026"}
Jul 16 23:11:15.247305 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:15.247254    2579 scope.go:117] "RemoveContainer" containerID="e63f03a8913b3310e873e0306afdcc0d6c56bddf07690b5a7ca4778b5541c27c"
Jul 16 23:11:15.247486 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:15.247473    2579 scope.go:117] "RemoveContainer" containerID="500af4646c3cd2f0758baed56fc49801d8d0d1a0ba19bf2c55907fdca80d4026"
Jul 16 23:11:15.247667 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:15.247650    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d_odh-ai-gateway-infra(d3a91367-b462-4ea9-a7ff-2c55b7348416)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416"
Jul 16 23:11:16.273873 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:16.273840    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:11:16.396553 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:16.396533    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:11:16.454218 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:16.454197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca\") pod \"d3a91367-b462-4ea9-a7ff-2c55b7348416\" (UID: \"d3a91367-b462-4ea9-a7ff-2c55b7348416\") "
Jul 16 23:11:16.454497 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:16.454474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3a91367-b462-4ea9-a7ff-2c55b7348416" (UID: "d3a91367-b462-4ea9-a7ff-2c55b7348416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:16.554978 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:16.554904    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a91367-b462-4ea9-a7ff-2c55b7348416-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:11:17.259549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:17.259516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d" event={"ID":"d3a91367-b462-4ea9-a7ff-2c55b7348416","Type":"ContainerDied","Data":"a3fd503240e10bea44952c03c90168e2ac104bb92a87ea2836be54d1a567fcad"}
Jul 16 23:11:17.259549 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:17.259544    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"
Jul 16 23:11:17.259805 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:17.259575    2579 scope.go:117] "RemoveContainer" containerID="500af4646c3cd2f0758baed56fc49801d8d0d1a0ba19bf2c55907fdca80d4026"
Jul 16 23:11:17.276049 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:17.276028    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:11:17.283295 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:17.283275    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vdv8d"]
Jul 16 23:11:18.466331 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:18.466297    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" path="/var/lib/kubelet/pods/d3a91367-b462-4ea9-a7ff-2c55b7348416/volumes"
Jul 16 23:11:19.376262 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376234    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:11:19.376594 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.376636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376595    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.376636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376611    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376617    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376623    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376636 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376629    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376640    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376646    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376655    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376660    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376724    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376741    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376747    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.376843 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.376753    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:19.381494 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.381476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.383835 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.383812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"openshift-service-ca.crt\""
Jul 16 23:11:19.384546 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.384524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"rate-test-model-61f04e-kserve-self-signed-certs\""
Jul 16 23:11:19.384546 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.384541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"default-dockercfg-rc8hh\""
Jul 16 23:11:19.384697 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.384556    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-61f04e\"/\"kube-root-ca.crt\""
Jul 16 23:11:19.389722 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.389689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:11:19.478487 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmj2w\" (UniqueName: \"kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.478851 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.478851 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.478851 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.478851 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.478851 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.478659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579646 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmj2w\" (UniqueName: \"kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.579926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.579886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.580070 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.580007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.580196 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.580177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.580266 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.580244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.581918 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.581899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.582226 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.582204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.588153 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.588134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmj2w\" (UniqueName: \"kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w\") pod \"rate-test-model-61f04e-kserve-748bd99dd7-td9fs\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") " pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.691616 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.691541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:19.821838 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:19.821810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:11:19.823828 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:11:19.823798    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ced6632_8a3e_494c_a4cf_309281351557.slice/crio-b128677af3554b9325680e47b1cbebfc1762719a5031546513d3702001155464 WatchSource:0}: Error finding container b128677af3554b9325680e47b1cbebfc1762719a5031546513d3702001155464: Status 404 returned error can't find the container with id b128677af3554b9325680e47b1cbebfc1762719a5031546513d3702001155464
Jul 16 23:11:20.274520 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:20.274491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" event={"ID":"8ced6632-8a3e-494c-a4cf-309281351557","Type":"ContainerStarted","Data":"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"}
Jul 16 23:11:20.274697 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:20.274529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" event={"ID":"8ced6632-8a3e-494c-a4cf-309281351557","Type":"ContainerStarted","Data":"b128677af3554b9325680e47b1cbebfc1762719a5031546513d3702001155464"}
Jul 16 23:11:20.291355 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:20.291303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" podStartSLOduration=1.2912891229999999 podStartE2EDuration="1.291289123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:11:20.290185537 +0000 UTC m=+3242.358508595" watchObservedRunningTime="2026-07-16 23:11:20.291289123 +0000 UTC m=+3242.359612182"
Jul 16 23:11:27.456466 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:27.456438    2579 scope.go:117] "RemoveContainer" containerID="53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc"
Jul 16 23:11:28.312153 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:28.312126    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerID="312cc697a1cb120ba14e15f6e9fe8fc41c53ee3ec3e86c1c55e24ed8d5740f4a" exitCode=6
Jul 16 23:11:28.312312 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:28.312202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerDied","Data":"312cc697a1cb120ba14e15f6e9fe8fc41c53ee3ec3e86c1c55e24ed8d5740f4a"}
Jul 16 23:11:28.312312 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:28.312266    2579 scope.go:117] "RemoveContainer" containerID="53b3e60257ffe2526700c7398dba9da1a55eb491130483e6813098488879dbbc"
Jul 16 23:11:28.312571 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:28.312553    2579 scope.go:117] "RemoveContainer" containerID="312cc697a1cb120ba14e15f6e9fe8fc41c53ee3ec3e86c1c55e24ed8d5740f4a"
Jul 16 23:11:28.312814 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:28.312795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-6zsmx_odh-ai-gateway-infra(7b68c497-ef67-4fda-b0a9-e8488ee5b086)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086"
Jul 16 23:11:29.338827 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.338791    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:29.470176 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.470153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:29.563483 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.563449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca\") pod \"7b68c497-ef67-4fda-b0a9-e8488ee5b086\" (UID: \"7b68c497-ef67-4fda-b0a9-e8488ee5b086\") "
Jul 16 23:11:29.563873 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.563849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b68c497-ef67-4fda-b0a9-e8488ee5b086" (UID: "7b68c497-ef67-4fda-b0a9-e8488ee5b086"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:29.664110 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.664040    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b68c497-ef67-4fda-b0a9-e8488ee5b086-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:11:29.691944 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.691922    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:29.692053 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.691968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:29.704217 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:29.704195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:30.324906 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.324878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"
Jul 16 23:11:30.325107 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.324920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx" event={"ID":"7b68c497-ef67-4fda-b0a9-e8488ee5b086","Type":"ContainerDied","Data":"b533671949adc872563971506639c77635472b9917015f98ca89b7f5aae2661f"}
Jul 16 23:11:30.325107 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.324979    2579 scope.go:117] "RemoveContainer" containerID="312cc697a1cb120ba14e15f6e9fe8fc41c53ee3ec3e86c1c55e24ed8d5740f4a"
Jul 16 23:11:30.337746 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.337730    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:11:30.350179 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.350153    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:30.360737 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.360716    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-6zsmx"]
Jul 16 23:11:30.461850 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:30.461823    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" path="/var/lib/kubelet/pods/7b68c497-ef67-4fda-b0a9-e8488ee5b086/volumes"
Jul 16 23:11:31.097201 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:31.097508 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097496    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097509    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097523    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097536    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097541    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097559    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097617    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa1b63d-e03e-4087-8681-5d088f7a1838" containerName="revoke-keys"
Jul 16 23:11:31.097824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097624    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097633    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a91367-b462-4ea9-a7ff-2c55b7348416" containerName="revoke-keys"
Jul 16 23:11:31.097824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097639    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.097824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.097646    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b68c497-ef67-4fda-b0a9-e8488ee5b086" containerName="revoke-keys"
Jul 16 23:11:31.101994 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.101977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:31.104253 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.104232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:11:31.108937 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.108912    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:31.174054 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.174025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9\" (UID: \"3f4aa0ea-9446-4813-9757-a513af57b356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:31.274556 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.274527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9\" (UID: \"3f4aa0ea-9446-4813-9757-a513af57b356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:31.275141 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.275125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9\" (UID: \"3f4aa0ea-9446-4813-9757-a513af57b356\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:31.414401 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.414337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:31.542943 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:31.542917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:31.544732 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:11:31.544693    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4aa0ea_9446_4813_9757_a513af57b356.slice/crio-28f505d82b3c6388930c41f34d30e45f6e404a49e7ba8f4f7979269b991f21b2 WatchSource:0}: Error finding container 28f505d82b3c6388930c41f34d30e45f6e404a49e7ba8f4f7979269b991f21b2: Status 404 returned error can't find the container with id 28f505d82b3c6388930c41f34d30e45f6e404a49e7ba8f4f7979269b991f21b2
Jul 16 23:11:32.337946 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:32.337917    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f4aa0ea-9446-4813-9757-a513af57b356" containerID="c2fb4171556d51fdda78a65f0af16bce43a69fa874048e6c6c24bb573bc5d7df" exitCode=6
Jul 16 23:11:32.338168 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:32.337994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" event={"ID":"3f4aa0ea-9446-4813-9757-a513af57b356","Type":"ContainerDied","Data":"c2fb4171556d51fdda78a65f0af16bce43a69fa874048e6c6c24bb573bc5d7df"}
Jul 16 23:11:32.338168 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:32.338046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" event={"ID":"3f4aa0ea-9446-4813-9757-a513af57b356","Type":"ContainerStarted","Data":"28f505d82b3c6388930c41f34d30e45f6e404a49e7ba8f4f7979269b991f21b2"}
Jul 16 23:11:32.338168 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:32.338115    2579 scope.go:117] "RemoveContainer" containerID="c2fb4171556d51fdda78a65f0af16bce43a69fa874048e6c6c24bb573bc5d7df"
Jul 16 23:11:33.344453 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:33.344426    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f4aa0ea-9446-4813-9757-a513af57b356" containerID="9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe" exitCode=6
Jul 16 23:11:33.344874 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:33.344507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" event={"ID":"3f4aa0ea-9446-4813-9757-a513af57b356","Type":"ContainerDied","Data":"9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe"}
Jul 16 23:11:33.344874 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:33.344560    2579 scope.go:117] "RemoveContainer" containerID="c2fb4171556d51fdda78a65f0af16bce43a69fa874048e6c6c24bb573bc5d7df"
Jul 16 23:11:33.344874 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:33.344672    2579 scope.go:117] "RemoveContainer" containerID="9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe"
Jul 16 23:11:33.344874 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:33.344860    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9_odh-ai-gateway-infra(3f4aa0ea-9446-4813-9757-a513af57b356)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" podUID="3f4aa0ea-9446-4813-9757-a513af57b356"
Jul 16 23:11:34.351834 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:34.351805    2579 scope.go:117] "RemoveContainer" containerID="9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe"
Jul 16 23:11:34.352212 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:34.351966    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9_odh-ai-gateway-infra(3f4aa0ea-9446-4813-9757-a513af57b356)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" podUID="3f4aa0ea-9446-4813-9757-a513af57b356"
Jul 16 23:11:45.456810 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:45.456775    2579 scope.go:117] "RemoveContainer" containerID="9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe"
Jul 16 23:11:46.408774 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:46.408672    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f4aa0ea-9446-4813-9757-a513af57b356" containerID="d31fa1028b8a59208859447e76829376beb390275f36adac2f3b51f6748781d2" exitCode=6
Jul 16 23:11:46.408774 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:46.408736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" event={"ID":"3f4aa0ea-9446-4813-9757-a513af57b356","Type":"ContainerDied","Data":"d31fa1028b8a59208859447e76829376beb390275f36adac2f3b51f6748781d2"}
Jul 16 23:11:46.408974 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:46.408797    2579 scope.go:117] "RemoveContainer" containerID="9c1845a0eb087c01df878e979befeee3a1d34dfcdb988e516bdf9eeb331c5bbe"
Jul 16 23:11:46.409094 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:46.409068    2579 scope.go:117] "RemoveContainer" containerID="d31fa1028b8a59208859447e76829376beb390275f36adac2f3b51f6748781d2"
Jul 16 23:11:46.409269 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:46.409247    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9_odh-ai-gateway-infra(3f4aa0ea-9446-4813-9757-a513af57b356)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" podUID="3f4aa0ea-9446-4813-9757-a513af57b356"
Jul 16 23:11:47.433697 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:47.433657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:47.572649 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:47.572623    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:47.705254 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:47.705180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca\") pod \"3f4aa0ea-9446-4813-9757-a513af57b356\" (UID: \"3f4aa0ea-9446-4813-9757-a513af57b356\") "
Jul 16 23:11:47.705505 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:47.705484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f4aa0ea-9446-4813-9757-a513af57b356" (UID: "3f4aa0ea-9446-4813-9757-a513af57b356"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:11:47.806536 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:47.806510    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f4aa0ea-9446-4813-9757-a513af57b356-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:11:48.421612 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.421581    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"
Jul 16 23:11:48.421926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.421624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9" event={"ID":"3f4aa0ea-9446-4813-9757-a513af57b356","Type":"ContainerDied","Data":"28f505d82b3c6388930c41f34d30e45f6e404a49e7ba8f4f7979269b991f21b2"}
Jul 16 23:11:48.421926 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.421682    2579 scope.go:117] "RemoveContainer" containerID="d31fa1028b8a59208859447e76829376beb390275f36adac2f3b51f6748781d2"
Jul 16 23:11:48.443452 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.443383    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:48.445277 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.445256    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-jpbr9"]
Jul 16 23:11:48.461681 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:48.461653    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" path="/var/lib/kubelet/pods/3f4aa0ea-9446-4813-9757-a513af57b356/volumes"
Jul 16 23:11:52.861359 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861330    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:11:52.861739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861679    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.861739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861691    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.861739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.861739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861738    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.861882 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861800    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.861882 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.861808    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:11:52.866202 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.866185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:11:52.868895 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.868875    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:11:52.871036 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:52.871006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:11:53.050578 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.050546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl\" (UID: \"caac3124-ada6-4c07-baa3-345d790966fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:11:53.151920 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.151852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl\" (UID: \"caac3124-ada6-4c07-baa3-345d790966fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:11:53.152431 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.152410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl\" (UID: \"caac3124-ada6-4c07-baa3-345d790966fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:11:53.177837 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.177817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:11:53.306777 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.306754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:11:53.309048 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:11:53.309022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac3124_ada6_4c07_baa3_345d790966fb.slice/crio-4d52d1f56924a0c401f8876b4c1400cdab8cf6455259555849993132d2f585b8 WatchSource:0}: Error finding container 4d52d1f56924a0c401f8876b4c1400cdab8cf6455259555849993132d2f585b8: Status 404 returned error can't find the container with id 4d52d1f56924a0c401f8876b4c1400cdab8cf6455259555849993132d2f585b8
Jul 16 23:11:53.446722 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:53.446645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" event={"ID":"caac3124-ada6-4c07-baa3-345d790966fb","Type":"ContainerStarted","Data":"4d52d1f56924a0c401f8876b4c1400cdab8cf6455259555849993132d2f585b8"}
Jul 16 23:11:54.452986 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:54.452950    2579 generic.go:356] "Generic (PLEG): container finished" podID="caac3124-ada6-4c07-baa3-345d790966fb" containerID="ebb19e83c945fb63fbc7f83fe4adfc77842dfbb6e848d08924a6b14b016c5353" exitCode=6
Jul 16 23:11:54.453374 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:54.453038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" event={"ID":"caac3124-ada6-4c07-baa3-345d790966fb","Type":"ContainerDied","Data":"ebb19e83c945fb63fbc7f83fe4adfc77842dfbb6e848d08924a6b14b016c5353"}
Jul 16 23:11:54.453374 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:54.453168    2579 scope.go:117] "RemoveContainer" containerID="ebb19e83c945fb63fbc7f83fe4adfc77842dfbb6e848d08924a6b14b016c5353"
Jul 16 23:11:55.459418 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:55.459393    2579 generic.go:356] "Generic (PLEG): container finished" podID="caac3124-ada6-4c07-baa3-345d790966fb" containerID="33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e" exitCode=6
Jul 16 23:11:55.459777 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:55.459482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" event={"ID":"caac3124-ada6-4c07-baa3-345d790966fb","Type":"ContainerDied","Data":"33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e"}
Jul 16 23:11:55.459777 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:55.459542    2579 scope.go:117] "RemoveContainer" containerID="ebb19e83c945fb63fbc7f83fe4adfc77842dfbb6e848d08924a6b14b016c5353"
Jul 16 23:11:55.459777 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:55.459640    2579 scope.go:117] "RemoveContainer" containerID="33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e"
Jul 16 23:11:55.459907 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:55.459833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl_odh-ai-gateway-infra(caac3124-ada6-4c07-baa3-345d790966fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" podUID="caac3124-ada6-4c07-baa3-345d790966fb"
Jul 16 23:11:56.464847 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:11:56.464820    2579 scope.go:117] "RemoveContainer" containerID="33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e"
Jul 16 23:11:56.465223 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:11:56.464968    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl_odh-ai-gateway-infra(caac3124-ada6-4c07-baa3-345d790966fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" podUID="caac3124-ada6-4c07-baa3-345d790966fb"
Jul 16 23:12:08.774355 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:08.774326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:12:10.457053 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:10.457018    2579 scope.go:117] "RemoveContainer" containerID="33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e"
Jul 16 23:12:11.540744 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:11.540683    2579 generic.go:356] "Generic (PLEG): container finished" podID="caac3124-ada6-4c07-baa3-345d790966fb" containerID="1fdedc692d8d81f2e53d4ee75ab1fc8da4eb2feff0fa003c320f34e508813b67" exitCode=6
Jul 16 23:12:11.540744 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:11.540737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" event={"ID":"caac3124-ada6-4c07-baa3-345d790966fb","Type":"ContainerDied","Data":"1fdedc692d8d81f2e53d4ee75ab1fc8da4eb2feff0fa003c320f34e508813b67"}
Jul 16 23:12:11.541208 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:11.540799    2579 scope.go:117] "RemoveContainer" containerID="33a98a3c31254f03da94d19e9db410e33fb0fc59e56ad3dde5c73eae6651949e"
Jul 16 23:12:11.541208 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:11.541028    2579 scope.go:117] "RemoveContainer" containerID="1fdedc692d8d81f2e53d4ee75ab1fc8da4eb2feff0fa003c320f34e508813b67"
Jul 16 23:12:11.541298 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:11.541229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl_odh-ai-gateway-infra(caac3124-ada6-4c07-baa3-345d790966fb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" podUID="caac3124-ada6-4c07-baa3-345d790966fb"
Jul 16 23:12:12.566244 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:12.566213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:12:12.702558 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:12.702535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:12:12.803688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:12.803661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca\") pod \"caac3124-ada6-4c07-baa3-345d790966fb\" (UID: \"caac3124-ada6-4c07-baa3-345d790966fb\") "
Jul 16 23:12:12.803969 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:12.803949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caac3124-ada6-4c07-baa3-345d790966fb" (UID: "caac3124-ada6-4c07-baa3-345d790966fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:12.905267 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:12.905207    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caac3124-ada6-4c07-baa3-345d790966fb-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:13.552070 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:13.552040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"
Jul 16 23:12:13.552268 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:13.552084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl" event={"ID":"caac3124-ada6-4c07-baa3-345d790966fb","Type":"ContainerDied","Data":"4d52d1f56924a0c401f8876b4c1400cdab8cf6455259555849993132d2f585b8"}
Jul 16 23:12:13.552268 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:13.552145    2579 scope.go:117] "RemoveContainer" containerID="1fdedc692d8d81f2e53d4ee75ab1fc8da4eb2feff0fa003c320f34e508813b67"
Jul 16 23:12:13.573185 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:13.573163    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:12:13.578799 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:13.578778    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-77kdl"]
Jul 16 23:12:14.462257 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:14.462225    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caac3124-ada6-4c07-baa3-345d790966fb" path="/var/lib/kubelet/pods/caac3124-ada6-4c07-baa3-345d790966fb/volumes"
Jul 16 23:12:21.148755 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.148717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:21.149317 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149244    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:12:21.149317 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149264    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:12:21.149317 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149287    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149317 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149295    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149321    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149330    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149402    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149417    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149427    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:21.149539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.149437    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4aa0ea-9446-4813-9757-a513af57b356" containerName="revoke-keys"
Jul 16 23:12:21.154312 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.154290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:21.157236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.157214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:12:21.159125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.159102    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:21.273669 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.273647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k\" (UID: \"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:21.374168 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.374147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k\" (UID: \"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:21.374677 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.374659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k\" (UID: \"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:21.465894 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.465833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:21.592648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:21.592624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:21.595005 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:12:21.594978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096a7fa8_bc14_46ee_b6d8_cd9dbe9a0832.slice/crio-d3d8b61a8128300835c17fbc436db98ce35e91313375365d8e1e8c779c069704 WatchSource:0}: Error finding container d3d8b61a8128300835c17fbc436db98ce35e91313375365d8e1e8c779c069704: Status 404 returned error can't find the container with id d3d8b61a8128300835c17fbc436db98ce35e91313375365d8e1e8c779c069704
Jul 16 23:12:22.594329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:22.594294    2579 generic.go:356] "Generic (PLEG): container finished" podID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerID="8c1cb731efdc4ff6873aa4d2ceca0d81e813977dedb8f431642cf222b201d52f" exitCode=6
Jul 16 23:12:22.594688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:22.594334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerDied","Data":"8c1cb731efdc4ff6873aa4d2ceca0d81e813977dedb8f431642cf222b201d52f"}
Jul 16 23:12:22.594688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:22.594381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerStarted","Data":"d3d8b61a8128300835c17fbc436db98ce35e91313375365d8e1e8c779c069704"}
Jul 16 23:12:22.594688 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:22.594579    2579 scope.go:117] "RemoveContainer" containerID="8c1cb731efdc4ff6873aa4d2ceca0d81e813977dedb8f431642cf222b201d52f"
Jul 16 23:12:23.601385 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:23.601360    2579 generic.go:356] "Generic (PLEG): container finished" podID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerID="04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023" exitCode=6
Jul 16 23:12:23.601833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:23.601449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerDied","Data":"04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023"}
Jul 16 23:12:23.601833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:23.601521    2579 scope.go:117] "RemoveContainer" containerID="8c1cb731efdc4ff6873aa4d2ceca0d81e813977dedb8f431642cf222b201d52f"
Jul 16 23:12:23.601833 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:23.601619    2579 scope.go:117] "RemoveContainer" containerID="04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023"
Jul 16 23:12:23.601833 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:23.601797    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k_odh-ai-gateway-infra(096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832"
Jul 16 23:12:23.794070 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:23.794033    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" podUID="8ced6632-8a3e-494c-a4cf-309281351557" containerName="main" containerID="cri-o://8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74" gracePeriod=45
Jul 16 23:12:24.047122 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.047096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:12:24.196287 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196222    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmj2w\" (UniqueName: \"kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196475 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home\") pod \"8ced6632-8a3e-494c-a4cf-309281351557\" (UID: \"8ced6632-8a3e-494c-a4cf-309281351557\") "
Jul 16 23:12:24.196771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:24.196771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:24.196771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.196726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home" (OuterVolumeSpecName: "home") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:24.198393 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.198371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm" (OuterVolumeSpecName: "dshm") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 23:12:24.198393 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.198388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w" (OuterVolumeSpecName: "kube-api-access-tmj2w") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "kube-api-access-tmj2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:12:24.198508 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.198377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ced6632-8a3e-494c-a4cf-309281351557" (UID: "8ced6632-8a3e-494c-a4cf-309281351557"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 23:12:24.297176 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297149    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-model-cache\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.297176 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297173    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmj2w\" (UniqueName: \"kubernetes.io/projected/8ced6632-8a3e-494c-a4cf-309281351557-kube-api-access-tmj2w\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.297319 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297187    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ced6632-8a3e-494c-a4cf-309281351557-tls-certs\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.297319 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297202    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-tmp-dir\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.297319 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297213    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-home\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.297319 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.297224    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ced6632-8a3e-494c-a4cf-309281351557-dshm\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:24.614377 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.614350    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ced6632-8a3e-494c-a4cf-309281351557" containerID="8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74" exitCode=0
Jul 16 23:12:24.614840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.614408    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"
Jul 16 23:12:24.614840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.614423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" event={"ID":"8ced6632-8a3e-494c-a4cf-309281351557","Type":"ContainerDied","Data":"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"}
Jul 16 23:12:24.614840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.614460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs" event={"ID":"8ced6632-8a3e-494c-a4cf-309281351557","Type":"ContainerDied","Data":"b128677af3554b9325680e47b1cbebfc1762719a5031546513d3702001155464"}
Jul 16 23:12:24.614840 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.614480    2579 scope.go:117] "RemoveContainer" containerID="8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"
Jul 16 23:12:24.616882 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.616653    2579 scope.go:117] "RemoveContainer" containerID="04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023"
Jul 16 23:12:24.616882 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:24.616865    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k_odh-ai-gateway-infra(096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832"
Jul 16 23:12:24.623071 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.623053    2579 scope.go:117] "RemoveContainer" containerID="8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"
Jul 16 23:12:24.623311 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:24.623294    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74\": container with ID starting with 8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74 not found: ID does not exist" containerID="8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"
Jul 16 23:12:24.623363 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.623319    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74"} err="failed to get container status \"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74\": rpc error: code = NotFound desc = could not find container \"8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74\": container with ID starting with 8576fe84f37c80c5583cdfb147396e02c6bb34409c16b787dceb97e812728a74 not found: ID does not exist"
Jul 16 23:12:24.630878 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.630856    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:12:24.632554 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:24.632535    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-61f04e/rate-test-model-61f04e-kserve-748bd99dd7-td9fs"]
Jul 16 23:12:25.212521 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212491    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-140274772277552"]
Jul 16 23:12:25.212898 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212883    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ced6632-8a3e-494c-a4cf-309281351557" containerName="main"
Jul 16 23:12:25.212957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212900    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ced6632-8a3e-494c-a4cf-309281351557" containerName="main"
Jul 16 23:12:25.212957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212918    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:25.212957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212924    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="caac3124-ada6-4c07-baa3-345d790966fb" containerName="revoke-keys"
Jul 16 23:12:25.213065 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.212979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ced6632-8a3e-494c-a4cf-309281351557" containerName="main"
Jul 16 23:12:25.217238 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.217223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:25.220209 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.220187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jzfmx\""
Jul 16 23:12:25.221946 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.221918    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-140274772277552"]
Jul 16 23:12:25.304832 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.304805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6zd\" (UniqueName: \"kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd\") pod \"test-curl-7525-140274772277552\" (UID: \"7f9218b5-c454-4f4f-b075-c3080576e510\") " pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:25.405152 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.405125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6zd\" (UniqueName: \"kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd\") pod \"test-curl-7525-140274772277552\" (UID: \"7f9218b5-c454-4f4f-b075-c3080576e510\") " pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:25.413467 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.413448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6zd\" (UniqueName: \"kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd\") pod \"test-curl-7525-140274772277552\" (UID: \"7f9218b5-c454-4f4f-b075-c3080576e510\") " pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:25.528211 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.528162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:25.656162 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:25.656141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-140274772277552"]
Jul 16 23:12:25.658011 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:12:25.657979    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9218b5_c454_4f4f_b075_c3080576e510.slice/crio-2d023fbfd22a97d17543cc001ca76b4fdbee5302e72c8f832166d7e9643e9240 WatchSource:0}: Error finding container 2d023fbfd22a97d17543cc001ca76b4fdbee5302e72c8f832166d7e9643e9240: Status 404 returned error can't find the container with id 2d023fbfd22a97d17543cc001ca76b4fdbee5302e72c8f832166d7e9643e9240
Jul 16 23:12:26.462474 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:26.462448    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ced6632-8a3e-494c-a4cf-309281351557" path="/var/lib/kubelet/pods/8ced6632-8a3e-494c-a4cf-309281351557/volumes"
Jul 16 23:12:26.628873 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:26.628795    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f9218b5-c454-4f4f-b075-c3080576e510" containerID="c2884a8ccafc130f3b88cb1ef3a1ec9f5cb9219307cbd9a2ffc92b5e97104fa1" exitCode=0
Jul 16 23:12:26.628873 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:26.628847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-140274772277552" event={"ID":"7f9218b5-c454-4f4f-b075-c3080576e510","Type":"ContainerDied","Data":"c2884a8ccafc130f3b88cb1ef3a1ec9f5cb9219307cbd9a2ffc92b5e97104fa1"}
Jul 16 23:12:26.629047 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:26.628874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-140274772277552" event={"ID":"7f9218b5-c454-4f4f-b075-c3080576e510","Type":"ContainerStarted","Data":"2d023fbfd22a97d17543cc001ca76b4fdbee5302e72c8f832166d7e9643e9240"}
Jul 16 23:12:27.762291 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.762266    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:27.797665 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.797641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7525-140274772277552_7f9218b5-c454-4f4f-b075-c3080576e510/test-curl-7525-140274772277552/0.log"
Jul 16 23:12:27.818557 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.818535    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-140274772277552"]
Jul 16 23:12:27.822111 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.822092    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-140274772277552"]
Jul 16 23:12:27.827534 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.827515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp6zd\" (UniqueName: \"kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd\") pod \"7f9218b5-c454-4f4f-b075-c3080576e510\" (UID: \"7f9218b5-c454-4f4f-b075-c3080576e510\") "
Jul 16 23:12:27.829614 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.829589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd" (OuterVolumeSpecName: "kube-api-access-lp6zd") pod "7f9218b5-c454-4f4f-b075-c3080576e510" (UID: "7f9218b5-c454-4f4f-b075-c3080576e510"). InnerVolumeSpecName "kube-api-access-lp6zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 23:12:27.928645 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.928581    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6zd\" (UniqueName: \"kubernetes.io/projected/7f9218b5-c454-4f4f-b075-c3080576e510-kube-api-access-lp6zd\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:27.998172 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.998145    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"]
Jul 16 23:12:27.998480 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.998468    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f9218b5-c454-4f4f-b075-c3080576e510" containerName="test-curl-7525-140274772277552"
Jul 16 23:12:27.998529 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.998482    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f9218b5-c454-4f4f-b075-c3080576e510" containerName="test-curl-7525-140274772277552"
Jul 16 23:12:27.998578 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:27.998568    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f9218b5-c454-4f4f-b075-c3080576e510" containerName="test-curl-7525-140274772277552"
Jul 16 23:12:28.001524 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.001510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:28.007733 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.007667    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"]
Jul 16 23:12:28.129757 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.129702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp\" (UID: \"99877f51-cdd0-4b12-8be7-6ee584b42e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:28.230196 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.230137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp\" (UID: \"99877f51-cdd0-4b12-8be7-6ee584b42e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:28.230680 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.230664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp\" (UID: \"99877f51-cdd0-4b12-8be7-6ee584b42e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:28.312877 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.312853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:28.444402 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.444378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"]
Jul 16 23:12:28.446076 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:12:28.446052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99877f51_cdd0_4b12_8be7_6ee584b42e56.slice/crio-a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2 WatchSource:0}: Error finding container a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2: Status 404 returned error can't find the container with id a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2
Jul 16 23:12:28.461968 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.461946    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f9218b5-c454-4f4f-b075-c3080576e510" path="/var/lib/kubelet/pods/7f9218b5-c454-4f4f-b075-c3080576e510/volumes"
Jul 16 23:12:28.648589 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.648553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp" event={"ID":"99877f51-cdd0-4b12-8be7-6ee584b42e56","Type":"ContainerStarted","Data":"a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2"}
Jul 16 23:12:28.650029 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.650008    2579 scope.go:117] "RemoveContainer" containerID="c2884a8ccafc130f3b88cb1ef3a1ec9f5cb9219307cbd9a2ffc92b5e97104fa1"
Jul 16 23:12:28.650144 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:28.650122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-140274772277552"
Jul 16 23:12:29.659870 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:29.659838    2579 generic.go:356] "Generic (PLEG): container finished" podID="99877f51-cdd0-4b12-8be7-6ee584b42e56" containerID="38447c0e7e617f5f6ca445c15a2e2094a8ac54cbdbf6ea3e0fa7727dda9a8651" exitCode=0
Jul 16 23:12:29.660238 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:29.659920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp" event={"ID":"99877f51-cdd0-4b12-8be7-6ee584b42e56","Type":"ContainerDied","Data":"38447c0e7e617f5f6ca445c15a2e2094a8ac54cbdbf6ea3e0fa7727dda9a8651"}
Jul 16 23:12:30.797591 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:30.797569    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:30.954385 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:30.954325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca\") pod \"99877f51-cdd0-4b12-8be7-6ee584b42e56\" (UID: \"99877f51-cdd0-4b12-8be7-6ee584b42e56\") "
Jul 16 23:12:30.954670 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:30.954644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99877f51-cdd0-4b12-8be7-6ee584b42e56" (UID: "99877f51-cdd0-4b12-8be7-6ee584b42e56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:31.055816 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:31.055797    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99877f51-cdd0-4b12-8be7-6ee584b42e56-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:31.669940 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:31.669912    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp"
Jul 16 23:12:31.670130 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:31.669955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp" event={"ID":"99877f51-cdd0-4b12-8be7-6ee584b42e56","Type":"ContainerDied","Data":"a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2"}
Jul 16 23:12:31.670130 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:31.670007    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a06ea8019b4cfa723f4335f9fd4eadbd48d564386f3a57c6285cdef142bd02e2"
Jul 16 23:12:35.456326 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:35.456295    2579 scope.go:117] "RemoveContainer" containerID="04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023"
Jul 16 23:12:35.690579 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:35.690554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerStarted","Data":"6be9e2a2f042fdfaa5ce2bab4788d4dfc9a12e2c227011c1993a9d962c527f7f"}
Jul 16 23:12:35.706087 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:35.706046    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" podStartSLOduration=14.706031579 podStartE2EDuration="14.706031579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:35.70519266 +0000 UTC m=+3317.773515717" watchObservedRunningTime="2026-07-16 23:12:35.706031579 +0000 UTC m=+3317.774354646"
Jul 16 23:12:36.696473 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:36.696440    2579 generic.go:356] "Generic (PLEG): container finished" podID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerID="6be9e2a2f042fdfaa5ce2bab4788d4dfc9a12e2c227011c1993a9d962c527f7f" exitCode=6
Jul 16 23:12:36.697125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:36.696492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerDied","Data":"6be9e2a2f042fdfaa5ce2bab4788d4dfc9a12e2c227011c1993a9d962c527f7f"}
Jul 16 23:12:36.697125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:36.696539    2579 scope.go:117] "RemoveContainer" containerID="04fd8c831d2e98b2542482edc5dd07407f3d32dbeb8ff9a6f1c3840e2416c023"
Jul 16 23:12:36.697125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:36.696799    2579 scope.go:117] "RemoveContainer" containerID="6be9e2a2f042fdfaa5ce2bab4788d4dfc9a12e2c227011c1993a9d962c527f7f"
Jul 16 23:12:36.697125 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:36.697045    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k_odh-ai-gateway-infra(096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832"
Jul 16 23:12:36.721669 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:36.721646    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:37.836076 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:37.836051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:38.007040 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.006969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca\") pod \"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832\" (UID: \"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832\") "
Jul 16 23:12:38.007303 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.007281    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" (UID: "096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:12:38.108569 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.108524    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:12:38.708783 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.708752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k" event={"ID":"096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832","Type":"ContainerDied","Data":"d3d8b61a8128300835c17fbc436db98ce35e91313375365d8e1e8c779c069704"}
Jul 16 23:12:38.708963 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.708811    2579 scope.go:117] "RemoveContainer" containerID="6be9e2a2f042fdfaa5ce2bab4788d4dfc9a12e2c227011c1993a9d962c527f7f"
Jul 16 23:12:38.708963 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.708825    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"
Jul 16 23:12:38.723987 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.723938    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:38.729116 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:38.729087    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-v8j5k"]
Jul 16 23:12:40.462124 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:40.462049    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" path="/var/lib/kubelet/pods/096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832/volumes"
Jul 16 23:12:47.574260 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574561    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574572    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574593    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574599    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574605    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99877f51-cdd0-4b12-8be7-6ee584b42e56" containerName="revoke-keys"
Jul 16 23:12:47.574618 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574610    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="99877f51-cdd0-4b12-8be7-6ee584b42e56" containerName="revoke-keys"
Jul 16 23:12:47.574860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574662    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574671    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="99877f51-cdd0-4b12-8be7-6ee584b42e56" containerName="revoke-keys"
Jul 16 23:12:47.574860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574680    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.574860 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.574686    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:47.577921 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.577904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:12:47.580186 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.580169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:12:47.585285 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.585117    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:12:47.682295 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.682250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vx475\" (UID: \"c3f33761-4788-4de3-81d0-b3a965678e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:12:47.782995 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.782967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vx475\" (UID: \"c3f33761-4788-4de3-81d0-b3a965678e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:12:47.783545 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.783528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-vx475\" (UID: \"c3f33761-4788-4de3-81d0-b3a965678e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:12:47.889386 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:47.889324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:12:48.225097 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:48.225070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:12:48.227099 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:12:48.227065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f33761_4788_4de3_81d0_b3a965678e09.slice/crio-953de1824dcfda7fdda5814fe56a5db63d154811469b969c71b38af743830ad7 WatchSource:0}: Error finding container 953de1824dcfda7fdda5814fe56a5db63d154811469b969c71b38af743830ad7: Status 404 returned error can't find the container with id 953de1824dcfda7fdda5814fe56a5db63d154811469b969c71b38af743830ad7
Jul 16 23:12:48.761901 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:48.761870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerStarted","Data":"af48d9071df9591d2134862835485b0c421406196c1468e8056b2ea650c19518"}
Jul 16 23:12:48.762236 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:48.761918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerStarted","Data":"953de1824dcfda7fdda5814fe56a5db63d154811469b969c71b38af743830ad7"}
Jul 16 23:12:48.777668 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:48.777614    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" podStartSLOduration=1.777596186 podStartE2EDuration="1.777596186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:48.777337353 +0000 UTC m=+3330.845660406" watchObservedRunningTime="2026-07-16 23:12:48.777596186 +0000 UTC m=+3330.845919243"
Jul 16 23:12:49.767859 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:49.767825    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3f33761-4788-4de3-81d0-b3a965678e09" containerID="af48d9071df9591d2134862835485b0c421406196c1468e8056b2ea650c19518" exitCode=6
Jul 16 23:12:49.768302 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:49.767917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerDied","Data":"af48d9071df9591d2134862835485b0c421406196c1468e8056b2ea650c19518"}
Jul 16 23:12:49.768302 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:49.768069    2579 scope.go:117] "RemoveContainer" containerID="af48d9071df9591d2134862835485b0c421406196c1468e8056b2ea650c19518"
Jul 16 23:12:50.774461 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:50.774388    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3f33761-4788-4de3-81d0-b3a965678e09" containerID="2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c" exitCode=6
Jul 16 23:12:50.774872 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:50.774474    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerDied","Data":"2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c"}
Jul 16 23:12:50.774872 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:50.774534    2579 scope.go:117] "RemoveContainer" containerID="af48d9071df9591d2134862835485b0c421406196c1468e8056b2ea650c19518"
Jul 16 23:12:50.774872 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:50.774609    2579 scope.go:117] "RemoveContainer" containerID="2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c"
Jul 16 23:12:50.774872 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:50.774802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vx475_odh-ai-gateway-infra(c3f33761-4788-4de3-81d0-b3a965678e09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" podUID="c3f33761-4788-4de3-81d0-b3a965678e09"
Jul 16 23:12:51.781458 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:51.781430    2579 scope.go:117] "RemoveContainer" containerID="2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c"
Jul 16 23:12:51.781859 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:51.781597    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vx475_odh-ai-gateway-infra(c3f33761-4788-4de3-81d0-b3a965678e09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" podUID="c3f33761-4788-4de3-81d0-b3a965678e09"
Jul 16 23:12:52.907037 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:52.906999    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:12:52.907659 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:52.907342    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:52.907659 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:52.907352    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="096a7fa8-bc14-46ee-b6d8-cd9dbe9a0832" containerName="revoke-keys"
Jul 16 23:12:52.911683 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:52.911666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:12:52.917830 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:52.917802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:12:53.024450 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.024418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5\" (UID: \"479518eb-2ec2-48d2-bf4a-1b6c55b85f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:12:53.125326 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.125293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5\" (UID: \"479518eb-2ec2-48d2-bf4a-1b6c55b85f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:12:53.125914 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.125896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5\" (UID: \"479518eb-2ec2-48d2-bf4a-1b6c55b85f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:12:53.222605 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.222548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:12:53.348675 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.348652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:12:53.350806 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:12:53.350779    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479518eb_2ec2_48d2_bf4a_1b6c55b85f28.slice/crio-7a631747f4dc710d61f6bf4ae0b8298f4151dc8da0c7d778fea1775a894969e9 WatchSource:0}: Error finding container 7a631747f4dc710d61f6bf4ae0b8298f4151dc8da0c7d778fea1775a894969e9: Status 404 returned error can't find the container with id 7a631747f4dc710d61f6bf4ae0b8298f4151dc8da0c7d778fea1775a894969e9
Jul 16 23:12:53.793221 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.793134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerStarted","Data":"564a6f7f79db15db738bfe79e2585998172fbc148fae72040363f58d4becef4e"}
Jul 16 23:12:53.793221 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.793184    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerStarted","Data":"7a631747f4dc710d61f6bf4ae0b8298f4151dc8da0c7d778fea1775a894969e9"}
Jul 16 23:12:53.812011 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:53.811959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" podStartSLOduration=1.8119450559999999 podStartE2EDuration="1.811945056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:12:53.808812316 +0000 UTC m=+3335.877135370" watchObservedRunningTime="2026-07-16 23:12:53.811945056 +0000 UTC m=+3335.880268109"
Jul 16 23:12:54.798899 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:54.798860    2579 generic.go:356] "Generic (PLEG): container finished" podID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerID="564a6f7f79db15db738bfe79e2585998172fbc148fae72040363f58d4becef4e" exitCode=6
Jul 16 23:12:54.799282 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:54.798912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerDied","Data":"564a6f7f79db15db738bfe79e2585998172fbc148fae72040363f58d4becef4e"}
Jul 16 23:12:54.799282 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:54.799158    2579 scope.go:117] "RemoveContainer" containerID="564a6f7f79db15db738bfe79e2585998172fbc148fae72040363f58d4becef4e"
Jul 16 23:12:55.805564 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:55.805471    2579 generic.go:356] "Generic (PLEG): container finished" podID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerID="436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442" exitCode=6
Jul 16 23:12:55.806022 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:55.805582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerDied","Data":"436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442"}
Jul 16 23:12:55.806022 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:55.805640    2579 scope.go:117] "RemoveContainer" containerID="564a6f7f79db15db738bfe79e2585998172fbc148fae72040363f58d4becef4e"
Jul 16 23:12:55.806022 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:55.805806    2579 scope.go:117] "RemoveContainer" containerID="436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442"
Jul 16 23:12:55.806188 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:55.806031    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5_odh-ai-gateway-infra(479518eb-2ec2-48d2-bf4a-1b6c55b85f28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28"
Jul 16 23:12:56.813794 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:12:56.813770    2579 scope.go:117] "RemoveContainer" containerID="436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442"
Jul 16 23:12:56.814193 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:12:56.813989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5_odh-ai-gateway-infra(479518eb-2ec2-48d2-bf4a-1b6c55b85f28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28"
Jul 16 23:13:03.456961 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:03.456930    2579 scope.go:117] "RemoveContainer" containerID="2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c"
Jul 16 23:13:03.847611 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:03.847585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerStarted","Data":"b7ac44a31449b4f3eff0fcb044091c0164ee9eeef4a5573389dc0e3998c42b52"}
Jul 16 23:13:04.853911 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:04.853878    2579 generic.go:356] "Generic (PLEG): container finished" podID="c3f33761-4788-4de3-81d0-b3a965678e09" containerID="b7ac44a31449b4f3eff0fcb044091c0164ee9eeef4a5573389dc0e3998c42b52" exitCode=6
Jul 16 23:13:04.853911 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:04.853914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerDied","Data":"b7ac44a31449b4f3eff0fcb044091c0164ee9eeef4a5573389dc0e3998c42b52"}
Jul 16 23:13:04.854422 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:04.853961    2579 scope.go:117] "RemoveContainer" containerID="2ca299c690ade89f7eba7c5fbd9737b6b31d59d1433292b0d6a2fc770da3dc1c"
Jul 16 23:13:04.854422 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:04.854243    2579 scope.go:117] "RemoveContainer" containerID="b7ac44a31449b4f3eff0fcb044091c0164ee9eeef4a5573389dc0e3998c42b52"
Jul 16 23:13:04.854533 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:04.854438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-vx475_odh-ai-gateway-infra(c3f33761-4788-4de3-81d0-b3a965678e09)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" podUID="c3f33761-4788-4de3-81d0-b3a965678e09"
Jul 16 23:13:04.874266 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:04.874245    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:13:05.992135 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:05.992108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:13:06.025324 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.025299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca\") pod \"c3f33761-4788-4de3-81d0-b3a965678e09\" (UID: \"c3f33761-4788-4de3-81d0-b3a965678e09\") "
Jul 16 23:13:06.025586 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.025566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3f33761-4788-4de3-81d0-b3a965678e09" (UID: "c3f33761-4788-4de3-81d0-b3a965678e09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:06.126333 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.126278    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f33761-4788-4de3-81d0-b3a965678e09-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:13:06.865514 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.865491    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"
Jul 16 23:13:06.865514 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.865508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475" event={"ID":"c3f33761-4788-4de3-81d0-b3a965678e09","Type":"ContainerDied","Data":"953de1824dcfda7fdda5814fe56a5db63d154811469b969c71b38af743830ad7"}
Jul 16 23:13:06.865745 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.865561    2579 scope.go:117] "RemoveContainer" containerID="b7ac44a31449b4f3eff0fcb044091c0164ee9eeef4a5573389dc0e3998c42b52"
Jul 16 23:13:06.883564 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.883542    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:13:06.887581 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:06.887560    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-vx475"]
Jul 16 23:13:08.462591 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:08.462558    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" path="/var/lib/kubelet/pods/c3f33761-4788-4de3-81d0-b3a965678e09/volumes"
Jul 16 23:13:09.456536 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:09.456507    2579 scope.go:117] "RemoveContainer" containerID="436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442"
Jul 16 23:13:09.883853 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:09.883830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerStarted","Data":"3f78116cbc9e4edd8835821561ca95efbbf7a6ddbf3adeae6dfadf1055fdcb5e"}
Jul 16 23:13:10.890132 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:10.890098    2579 generic.go:356] "Generic (PLEG): container finished" podID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerID="3f78116cbc9e4edd8835821561ca95efbbf7a6ddbf3adeae6dfadf1055fdcb5e" exitCode=6
Jul 16 23:13:10.890582 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:10.890139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerDied","Data":"3f78116cbc9e4edd8835821561ca95efbbf7a6ddbf3adeae6dfadf1055fdcb5e"}
Jul 16 23:13:10.890582 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:10.890183    2579 scope.go:117] "RemoveContainer" containerID="436f6c733ca3fb2ed0bb0cddb1a43b63440d02cdb17c5ed2dc406753013c0442"
Jul 16 23:13:10.890582 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:10.890438    2579 scope.go:117] "RemoveContainer" containerID="3f78116cbc9e4edd8835821561ca95efbbf7a6ddbf3adeae6dfadf1055fdcb5e"
Jul 16 23:13:10.890804 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:10.890633    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5_odh-ai-gateway-infra(479518eb-2ec2-48d2-bf4a-1b6c55b85f28)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28"
Jul 16 23:13:10.912018 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:10.911994    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:13:11.182318 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182252    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:11.182604 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182591    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182606    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182619    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182626    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182633    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182638    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182845 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182700    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182845 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.182845 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.182729    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f33761-4788-4de3-81d0-b3a965678e09" containerName="revoke-keys"
Jul 16 23:13:11.186803 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.186785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:11.192309 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.192273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:11.265912 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.265888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct\" (UID: \"784bf5d6-e02c-4602-8fde-5b585bf230fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:11.367000 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.366976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct\" (UID: \"784bf5d6-e02c-4602-8fde-5b585bf230fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:11.367533 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.367517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct\" (UID: \"784bf5d6-e02c-4602-8fde-5b585bf230fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:11.499798 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.499747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:11.828857 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.828786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:11.830998 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:13:11.830967    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784bf5d6_e02c_4602_8fde_5b585bf230fd.slice/crio-48467c7356770c303a713879b9b83e7ad18a93db829270a65bfa5a2e7589581d WatchSource:0}: Error finding container 48467c7356770c303a713879b9b83e7ad18a93db829270a65bfa5a2e7589581d: Status 404 returned error can't find the container with id 48467c7356770c303a713879b9b83e7ad18a93db829270a65bfa5a2e7589581d
Jul 16 23:13:11.896350 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:11.896317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerStarted","Data":"48467c7356770c303a713879b9b83e7ad18a93db829270a65bfa5a2e7589581d"}
Jul 16 23:13:12.030699 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.030676    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:13:12.073251 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.073225    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca\") pod \"479518eb-2ec2-48d2-bf4a-1b6c55b85f28\" (UID: \"479518eb-2ec2-48d2-bf4a-1b6c55b85f28\") "
Jul 16 23:13:12.073539 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.073520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "479518eb-2ec2-48d2-bf4a-1b6c55b85f28" (UID: "479518eb-2ec2-48d2-bf4a-1b6c55b85f28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:12.174576 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.174552    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/479518eb-2ec2-48d2-bf4a-1b6c55b85f28-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:13:12.904163 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.904140    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"
Jul 16 23:13:12.904607 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.904181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5" event={"ID":"479518eb-2ec2-48d2-bf4a-1b6c55b85f28","Type":"ContainerDied","Data":"7a631747f4dc710d61f6bf4ae0b8298f4151dc8da0c7d778fea1775a894969e9"}
Jul 16 23:13:12.904607 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.904246    2579 scope.go:117] "RemoveContainer" containerID="3f78116cbc9e4edd8835821561ca95efbbf7a6ddbf3adeae6dfadf1055fdcb5e"
Jul 16 23:13:12.905953 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.905934    2579 generic.go:356] "Generic (PLEG): container finished" podID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerID="b414a1942b1030d0a9e5afc6121e7599379e630cfaec329f25036812a41de975" exitCode=6
Jul 16 23:13:12.906057 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.905985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerDied","Data":"b414a1942b1030d0a9e5afc6121e7599379e630cfaec329f25036812a41de975"}
Jul 16 23:13:12.906218 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.906201    2579 scope.go:117] "RemoveContainer" containerID="b414a1942b1030d0a9e5afc6121e7599379e630cfaec329f25036812a41de975"
Jul 16 23:13:12.921156 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.921130    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:13:12.926217 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:12.926197    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5cb4a56a-lskd5"]
Jul 16 23:13:13.913652 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:13.913579    2579 generic.go:356] "Generic (PLEG): container finished" podID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerID="cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8" exitCode=6
Jul 16 23:13:13.913652 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:13.913621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerDied","Data":"cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8"}
Jul 16 23:13:13.914131 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:13.913682    2579 scope.go:117] "RemoveContainer" containerID="b414a1942b1030d0a9e5afc6121e7599379e630cfaec329f25036812a41de975"
Jul 16 23:13:13.914131 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:13.913865    2579 scope.go:117] "RemoveContainer" containerID="cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8"
Jul 16 23:13:13.914131 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:13.914051    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct_odh-ai-gateway-infra(784bf5d6-e02c-4602-8fde-5b585bf230fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd"
Jul 16 23:13:14.461620 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:14.461591    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" path="/var/lib/kubelet/pods/479518eb-2ec2-48d2-bf4a-1b6c55b85f28/volumes"
Jul 16 23:13:14.920576 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:14.920546    2579 scope.go:117] "RemoveContainer" containerID="cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8"
Jul 16 23:13:14.920967 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:14.920760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct_odh-ai-gateway-infra(784bf5d6-e02c-4602-8fde-5b585bf230fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd"
Jul 16 23:13:20.447062 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:20.447030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-647bd46cf4-h7tlr_38787a79-6938-4f12-ba3d-044c2659e025/authorino/0.log"
Jul 16 23:13:25.657462 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:25.657432    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-28m6q_1a53e0ad-e5b4-400c-80e7-9b0b416a23c1/manager/0.log"
Jul 16 23:13:26.385544 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:26.385516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj_4927ca4e-c54d-47a0-9c70-30d1be286cfc/copy-manifests/0.log"
Jul 16 23:13:26.392385 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:26.392364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj_4927ca4e-c54d-47a0-9c70-30d1be286cfc/manager/0.log"
Jul 16 23:13:27.185516 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:27.185485    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-b9c909-9dphd_0791022f-f658-42ab-86e1-727188a4255b/revoke-keys/0.log"
Jul 16 23:13:27.291205 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:27.291178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-8c14db-svlpb_635a7f3d-b483-4d4e-bcfd-6788ed8873ca/revoke-keys/0.log"
Jul 16 23:13:27.508951 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:27.508881    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-30cb29-5f5cq_4c5899df-aecf-4da3-9b23-54c988a6f721/revoke-keys/0.log"
Jul 16 23:13:27.613452 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:27.613433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct_784bf5d6-e02c-4602-8fde-5b585bf230fd/revoke-keys/1.log"
Jul 16 23:13:27.930506 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:27.930473    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-b5738f-mfgmp_99877f51-cdd0-4b12-8be7-6ee584b42e56/revoke-keys/0.log"
Jul 16 23:13:28.140665 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:28.140628    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-s6k7g_b50e2c63-af4d-46c4-96d9-61505413be53/postgres/0.log"
Jul 16 23:13:29.379829 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:29.379796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-647bd46cf4-h7tlr_38787a79-6938-4f12-ba3d-044c2659e025/authorino/0.log"
Jul 16 23:13:29.456578 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:29.456550    2579 scope.go:117] "RemoveContainer" containerID="cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8"
Jul 16 23:13:29.991412 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:29.991333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerStarted","Data":"908eaea8b0637c951e439ca9fdddc97d9a89a057d5185fc1aa2db8a70ef054c1"}
Jul 16 23:13:30.006756 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.006692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" podStartSLOduration=19.006678602 podStartE2EDuration="19.006678602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:13:30.005550698 +0000 UTC m=+3372.073873755" watchObservedRunningTime="2026-07-16 23:13:30.006678602 +0000 UTC m=+3372.075001653"
Jul 16 23:13:30.865615 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.865592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b969c6b8c-kdw76_1d9a09f9-a705-4ca7-b9ab-002e598b7db3/kube-auth-proxy/0.log"
Jul 16 23:13:30.996995 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.996969    2579 generic.go:356] "Generic (PLEG): container finished" podID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerID="908eaea8b0637c951e439ca9fdddc97d9a89a057d5185fc1aa2db8a70ef054c1" exitCode=6
Jul 16 23:13:30.997146 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.997047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerDied","Data":"908eaea8b0637c951e439ca9fdddc97d9a89a057d5185fc1aa2db8a70ef054c1"}
Jul 16 23:13:30.997146 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.997117    2579 scope.go:117] "RemoveContainer" containerID="cf9030888acfc61afc5046e928252ab1809a97d7876ae6bf43cda9157b7b50d8"
Jul 16 23:13:30.997382 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:30.997370    2579 scope.go:117] "RemoveContainer" containerID="908eaea8b0637c951e439ca9fdddc97d9a89a057d5185fc1aa2db8a70ef054c1"
Jul 16 23:13:30.997551 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:30.997536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct_odh-ai-gateway-infra(784bf5d6-e02c-4602-8fde-5b585bf230fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd"
Jul 16 23:13:31.018415 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:31.018393    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:31.214589 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:31.214530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-7jm7k_765eb737-62ad-408f-bdc5-fa9dd064c524/payload-processing/0.log"
Jul 16 23:13:31.649576 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:31.649551    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-sj2j9_7332ab36-1b3b-4867-bb53-5b503d6fc5fa/main/0.log"
Jul 16 23:13:32.086057 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.086032    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-fmkr2_c42de968-bdaa-4a01-bf64-c1147381736e/main/0.log"
Jul 16 23:13:32.134082 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.134062    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:32.190285 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.190263    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-c27zx_5906fa26-ee75-4260-aecf-32245c75dd17/main/0.log"
Jul 16 23:13:32.244957 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.244936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca\") pod \"784bf5d6-e02c-4602-8fde-5b585bf230fd\" (UID: \"784bf5d6-e02c-4602-8fde-5b585bf230fd\") "
Jul 16 23:13:32.245246 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.245228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "784bf5d6-e02c-4602-8fde-5b585bf230fd" (UID: "784bf5d6-e02c-4602-8fde-5b585bf230fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:32.346030 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:32.346006    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/784bf5d6-e02c-4602-8fde-5b585bf230fd-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:13:33.008771 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:33.008737    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct" event={"ID":"784bf5d6-e02c-4602-8fde-5b585bf230fd","Type":"ContainerDied","Data":"48467c7356770c303a713879b9b83e7ad18a93db829270a65bfa5a2e7589581d"}
Jul 16 23:13:33.008935 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:33.008800    2579 scope.go:117] "RemoveContainer" containerID="908eaea8b0637c951e439ca9fdddc97d9a89a057d5185fc1aa2db8a70ef054c1"
Jul 16 23:13:33.008935 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:33.008752    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"
Jul 16 23:13:33.023529 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:33.023504    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:33.029662 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:33.029640    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bf38b381-ftgct"]
Jul 16 23:13:34.461606 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:34.461571    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" path="/var/lib/kubelet/pods/784bf5d6-e02c-4602-8fde-5b585bf230fd/volumes"
Jul 16 23:13:37.620739 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.620697    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621041    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621053    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621061    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621067    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621078    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621088 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621091    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621097    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621106    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621111    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621118    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621122    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621186    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.621313 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.621194    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:37.625590 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.625573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:37.628096 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.628073    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tqg8r\""
Jul 16 23:13:37.630433 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.630408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:37.686001 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.685975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-hskml\" (UID: \"9a39bd58-e1c2-4331-969c-903214d963a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:37.787023 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.786995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-hskml\" (UID: \"9a39bd58-e1c2-4331-969c-903214d963a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:37.787577 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.787562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-75e8dc20-hskml\" (UID: \"9a39bd58-e1c2-4331-969c-903214d963a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:37.937205 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:37.937128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:38.276308 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:38.276223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:38.278650 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:13:38.278622    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a39bd58_e1c2_4331_969c_903214d963a7.slice/crio-3f1d60d2fb39cb5540fcc66bc73dffdc6cf8d03e0a83569b3a3c3e13f10fb078 WatchSource:0}: Error finding container 3f1d60d2fb39cb5540fcc66bc73dffdc6cf8d03e0a83569b3a3c3e13f10fb078: Status 404 returned error can't find the container with id 3f1d60d2fb39cb5540fcc66bc73dffdc6cf8d03e0a83569b3a3c3e13f10fb078
Jul 16 23:13:39.039409 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:39.039377    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a39bd58-e1c2-4331-969c-903214d963a7" containerID="7f5702425fa041a7e0c0453b36b5c6d8cef283a741e567c9a42fe434aaf4ed58" exitCode=6
Jul 16 23:13:39.039796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:39.039414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerDied","Data":"7f5702425fa041a7e0c0453b36b5c6d8cef283a741e567c9a42fe434aaf4ed58"}
Jul 16 23:13:39.039796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:39.039451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerStarted","Data":"3f1d60d2fb39cb5540fcc66bc73dffdc6cf8d03e0a83569b3a3c3e13f10fb078"}
Jul 16 23:13:39.039796 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:39.039622    2579 scope.go:117] "RemoveContainer" containerID="7f5702425fa041a7e0c0453b36b5c6d8cef283a741e567c9a42fe434aaf4ed58"
Jul 16 23:13:40.045944 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.045914    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a39bd58-e1c2-4331-969c-903214d963a7" containerID="bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56" exitCode=6
Jul 16 23:13:40.046381 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.046001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerDied","Data":"bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56"}
Jul 16 23:13:40.046381 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.046071    2579 scope.go:117] "RemoveContainer" containerID="7f5702425fa041a7e0c0453b36b5c6d8cef283a741e567c9a42fe434aaf4ed58"
Jul 16 23:13:40.046381 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.046130    2579 scope.go:117] "RemoveContainer" containerID="bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56"
Jul 16 23:13:40.046381 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:40.046346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-hskml_odh-ai-gateway-infra(9a39bd58-e1c2-4331-969c-903214d963a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" podUID="9a39bd58-e1c2-4331-969c-903214d963a7"
Jul 16 23:13:40.262058 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.262038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6bwd5_78b06a78-a819-426c-8aee-a671d9a09e6b/global-pull-secret-syncer/0.log"
Jul 16 23:13:40.474107 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.474083    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v2w7b_fb7d03e2-0bf9-4bcf-84bb-fa7b7c8a0a2a/konnectivity-agent/0.log"
Jul 16 23:13:40.538743 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:40.538718    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-9.ec2.internal_f0b61d43fdb33aaaa624fc245668ccfb/haproxy/0.log"
Jul 16 23:13:41.051988 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:41.051961    2579 scope.go:117] "RemoveContainer" containerID="bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56"
Jul 16 23:13:41.052432 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:41.052167    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-hskml_odh-ai-gateway-infra(9a39bd58-e1c2-4331-969c-903214d963a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" podUID="9a39bd58-e1c2-4331-969c-903214d963a7"
Jul 16 23:13:45.415867 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:45.415830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-647bd46cf4-h7tlr_38787a79-6938-4f12-ba3d-044c2659e025/authorino/0.log"
Jul 16 23:13:47.991691 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:47.991602    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lgsx5_505ecd5f-91bc-419a-a6e9-59bf9386366e/kube-state-metrics/0.log"
Jul 16 23:13:48.010911 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.010890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lgsx5_505ecd5f-91bc-419a-a6e9-59bf9386366e/kube-rbac-proxy-main/0.log"
Jul 16 23:13:48.031451 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.031431    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lgsx5_505ecd5f-91bc-419a-a6e9-59bf9386366e/kube-rbac-proxy-self/0.log"
Jul 16 23:13:48.059670 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.059650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-86cdffd95b-ffbc4_85f6e921-1f22-4dfb-b4d1-b94bc8fe519c/metrics-server/0.log"
Jul 16 23:13:48.266776 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.266675    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrmrz_92defb58-8f20-433c-a0d1-dacc3f266ff6/node-exporter/0.log"
Jul 16 23:13:48.287125 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.287106    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrmrz_92defb58-8f20-433c-a0d1-dacc3f266ff6/kube-rbac-proxy/0.log"
Jul 16 23:13:48.307220 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.307204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hrmrz_92defb58-8f20-433c-a0d1-dacc3f266ff6/init-textfile/0.log"
Jul 16 23:13:48.432824 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.432798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/prometheus/0.log"
Jul 16 23:13:48.454104 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.454076    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/config-reloader/0.log"
Jul 16 23:13:48.475680 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.475595    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/thanos-sidecar/0.log"
Jul 16 23:13:48.496859 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.496838    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/kube-rbac-proxy-web/0.log"
Jul 16 23:13:48.521311 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.521260    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/kube-rbac-proxy/0.log"
Jul 16 23:13:48.542680 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.542662    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/kube-rbac-proxy-thanos/0.log"
Jul 16 23:13:48.565234 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.565218    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d75575e-b323-40ab-8e75-0d4177783b20/init-config-reloader/0.log"
Jul 16 23:13:48.595308 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.595284    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-m95pp_496a12e6-e349-44a8-be45-867744d20e65/prometheus-operator/0.log"
Jul 16 23:13:48.613913 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.613895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-m95pp_496a12e6-e349-44a8-be45-867744d20e65/kube-rbac-proxy/0.log"
Jul 16 23:13:48.751940 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.751920    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/thanos-query/0.log"
Jul 16 23:13:48.773149 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.773101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/kube-rbac-proxy-web/0.log"
Jul 16 23:13:48.805495 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.805476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/kube-rbac-proxy/0.log"
Jul 16 23:13:48.825072 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.825054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/prom-label-proxy/0.log"
Jul 16 23:13:48.845614 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.845597    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/kube-rbac-proxy-rules/0.log"
Jul 16 23:13:48.865762 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:48.865742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-f4fbcf599-v78kv_5e0f7999-0614-4373-a481-0e5f8ecedddb/kube-rbac-proxy-metrics/0.log"
Jul 16 23:13:49.535694 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.535660    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"]
Jul 16 23:13:49.536127 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.536113    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="784bf5d6-e02c-4602-8fde-5b585bf230fd" containerName="revoke-keys"
Jul 16 23:13:49.536172 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.536129    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="479518eb-2ec2-48d2-bf4a-1b6c55b85f28" containerName="revoke-keys"
Jul 16 23:13:49.539124 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.539109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:13:49.545366 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.545328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"]
Jul 16 23:13:49.680278 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.680257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-v2v7l\" (UID: \"98724f2f-4f90-426d-a8d4-f531159c4d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:13:49.781047 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.781021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-v2v7l\" (UID: \"98724f2f-4f90-426d-a8d4-f531159c4d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:13:49.781558 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.781541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-234edc78-v2v7l\" (UID: \"98724f2f-4f90-426d-a8d4-f531159c4d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:13:49.850089 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:49.850069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:13:50.179232 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.179211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"]
Jul 16 23:13:50.180935 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:13:50.180908    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98724f2f_4f90_426d_a8d4_f531159c4d56.slice/crio-4cf79c8729e8bcefc866f37cf956179386e403c151bffa515b33fd9f8dc8973a WatchSource:0}: Error finding container 4cf79c8729e8bcefc866f37cf956179386e403c151bffa515b33fd9f8dc8973a: Status 404 returned error can't find the container with id 4cf79c8729e8bcefc866f37cf956179386e403c151bffa515b33fd9f8dc8973a
Jul 16 23:13:50.783962 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.783898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"]
Jul 16 23:13:50.787413 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.787394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.789525 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.789502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvjp6\"/\"openshift-service-ca.crt\""
Jul 16 23:13:50.789670 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.789653    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wvjp6\"/\"default-dockercfg-sgxhw\""
Jul 16 23:13:50.789754 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.789687    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvjp6\"/\"kube-root-ca.crt\""
Jul 16 23:13:50.797537 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.797519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"]
Jul 16 23:13:50.888283 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.888257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-podres\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.888403 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.888297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpk5d\" (UniqueName: \"kubernetes.io/projected/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-kube-api-access-jpk5d\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.888403 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.888329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-proc\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.888500 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.888415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-sys\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.888500 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.888457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-lib-modules\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-podres\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpk5d\" (UniqueName: \"kubernetes.io/projected/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-kube-api-access-jpk5d\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-proc\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-sys\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989306 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-lib-modules\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989460 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-podres\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989460 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-proc\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989460 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-sys\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.989460 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.989404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-lib-modules\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:50.996501 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:50.996474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpk5d\" (UniqueName: \"kubernetes.io/projected/b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4-kube-api-access-jpk5d\") pod \"perf-node-gather-daemonset-wnbr7\" (UID: \"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4\") " pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:51.097244 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.097224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:51.100325 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.100306    2579 generic.go:356] "Generic (PLEG): container finished" podID="98724f2f-4f90-426d-a8d4-f531159c4d56" containerID="b18d4b944489ffdf0d631e43fe3b28ea32a4bdbff1f6e4839bce695892b3d31b" exitCode=6
Jul 16 23:13:51.100409 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.100387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" event={"ID":"98724f2f-4f90-426d-a8d4-f531159c4d56","Type":"ContainerDied","Data":"b18d4b944489ffdf0d631e43fe3b28ea32a4bdbff1f6e4839bce695892b3d31b"}
Jul 16 23:13:51.100480 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.100468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" event={"ID":"98724f2f-4f90-426d-a8d4-f531159c4d56","Type":"ContainerStarted","Data":"4cf79c8729e8bcefc866f37cf956179386e403c151bffa515b33fd9f8dc8973a"}
Jul 16 23:13:51.100638 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.100626    2579 scope.go:117] "RemoveContainer" containerID="b18d4b944489ffdf0d631e43fe3b28ea32a4bdbff1f6e4839bce695892b3d31b"
Jul 16 23:13:51.226251 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.226226    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-tnxm8_5f375af6-8bb1-4e3c-a8ac-ad6f6c8d5e53/download-server/0.log"
Jul 16 23:13:51.253503 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:51.253481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"]
Jul 16 23:13:51.255887 ip-10-0-141-9 kubenswrapper[2579]: W0716 23:13:51.255863    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4979e29_c1af_4cd6_8064_b4e5cbc5ecd4.slice/crio-9be3c5c8a505ab93981cf832158a9eb4ae12584b276628b780c0c480f01edf79 WatchSource:0}: Error finding container 9be3c5c8a505ab93981cf832158a9eb4ae12584b276628b780c0c480f01edf79: Status 404 returned error can't find the container with id 9be3c5c8a505ab93981cf832158a9eb4ae12584b276628b780c0c480f01edf79
Jul 16 23:13:52.110232 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.110200    2579 generic.go:356] "Generic (PLEG): container finished" podID="98724f2f-4f90-426d-a8d4-f531159c4d56" containerID="dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3" exitCode=6
Jul 16 23:13:52.110641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.110285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" event={"ID":"98724f2f-4f90-426d-a8d4-f531159c4d56","Type":"ContainerDied","Data":"dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3"}
Jul 16 23:13:52.110641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.110346    2579 scope.go:117] "RemoveContainer" containerID="b18d4b944489ffdf0d631e43fe3b28ea32a4bdbff1f6e4839bce695892b3d31b"
Jul 16 23:13:52.110641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.110518    2579 scope.go:117] "RemoveContainer" containerID="dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3"
Jul 16 23:13:52.110805 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:52.110742    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-v2v7l_odh-ai-gateway-infra(98724f2f-4f90-426d-a8d4-f531159c4d56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" podUID="98724f2f-4f90-426d-a8d4-f531159c4d56"
Jul 16 23:13:52.111635 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.111597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7" event={"ID":"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4","Type":"ContainerStarted","Data":"8e8157cd52bfcecbd65475f3f6c7ec9c326933b822461957ff935125187b8fa8"}
Jul 16 23:13:52.111635 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.111635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7" event={"ID":"b4979e29-c1af-4cd6-8064-b4e5cbc5ecd4","Type":"ContainerStarted","Data":"9be3c5c8a505ab93981cf832158a9eb4ae12584b276628b780c0c480f01edf79"}
Jul 16 23:13:52.111786 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.111755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:52.141648 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.141601    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7" podStartSLOduration=2.14158931 podStartE2EDuration="2.14158931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 23:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:13:52.140048619 +0000 UTC m=+3394.208371674" watchObservedRunningTime="2026-07-16 23:13:52.14158931 +0000 UTC m=+3394.209912363"
Jul 16 23:13:52.456907 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:52.456827    2579 scope.go:117] "RemoveContainer" containerID="bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56"
Jul 16 23:13:53.051093 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.051026    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t5smt_01d1b9d5-9451-4525-a2e1-5f52d140a07a/dns/0.log"
Jul 16 23:13:53.072872 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.072849    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t5smt_01d1b9d5-9451-4525-a2e1-5f52d140a07a/kube-rbac-proxy/0.log"
Jul 16 23:13:53.119098 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.119074    2579 scope.go:117] "RemoveContainer" containerID="dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3"
Jul 16 23:13:53.119463 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:53.119264    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-v2v7l_odh-ai-gateway-infra(98724f2f-4f90-426d-a8d4-f531159c4d56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" podUID="98724f2f-4f90-426d-a8d4-f531159c4d56"
Jul 16 23:13:53.120989 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.120964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerStarted","Data":"7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce"}
Jul 16 23:13:53.121230 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.121217    2579 scope.go:117] "RemoveContainer" containerID="7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce"
Jul 16 23:13:53.121377 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:53.121362    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-hskml_odh-ai-gateway-infra(9a39bd58-e1c2-4331-969c-903214d963a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" podUID="9a39bd58-e1c2-4331-969c-903214d963a7"
Jul 16 23:13:53.148837 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.148816    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkx9c_70592430-0a46-46b8-8650-2b3bfca5bb1e/dns-node-resolver/0.log"
Jul 16 23:13:53.739369 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:53.739336    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7w9cp_069f7c17-92a1-44ae-9039-3008ce6bda24/node-ca/0.log"
Jul 16 23:13:54.126574 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.126548    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a39bd58-e1c2-4331-969c-903214d963a7" containerID="7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce" exitCode=6
Jul 16 23:13:54.126992 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.126618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerDied","Data":"7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce"}
Jul 16 23:13:54.126992 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.126680    2579 scope.go:117] "RemoveContainer" containerID="bf9ee6ea5be059a8d6230e0451bbd42f6ebfc27e7ab82104caf2b9a34143bb56"
Jul 16 23:13:54.126992 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.126923    2579 scope.go:117] "RemoveContainer" containerID="7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce"
Jul 16 23:13:54.127142 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:13:54.127119    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-75e8dc20-hskml_odh-ai-gateway-infra(9a39bd58-e1c2-4331-969c-903214d963a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" podUID="9a39bd58-e1c2-4331-969c-903214d963a7"
Jul 16 23:13:54.155641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.155590    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:54.920939 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:54.920914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b969c6b8c-kdw76_1d9a09f9-a705-4ca7-b9ab-002e598b7db3/kube-auth-proxy/0.log"
Jul 16 23:13:55.066761 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.066733    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-7jm7k_765eb737-62ad-408f-bdc5-fa9dd064c524/payload-processing/0.log"
Jul 16 23:13:55.266273 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.266255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:55.427229 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.427195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca\") pod \"9a39bd58-e1c2-4331-969c-903214d963a7\" (UID: \"9a39bd58-e1c2-4331-969c-903214d963a7\") "
Jul 16 23:13:55.427526 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.427506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a39bd58-e1c2-4331-969c-903214d963a7" (UID: "9a39bd58-e1c2-4331-969c-903214d963a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:13:55.528023 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.527972    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a39bd58-e1c2-4331-969c-903214d963a7-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:13:55.841856 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:55.841830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t7wwr_484a6447-e587-4e1e-b292-3423d2e066cb/serve-healthcheck-canary/0.log"
Jul 16 23:13:56.139210 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.139146    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"
Jul 16 23:13:56.139348 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.139146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml" event={"ID":"9a39bd58-e1c2-4331-969c-903214d963a7","Type":"ContainerDied","Data":"3f1d60d2fb39cb5540fcc66bc73dffdc6cf8d03e0a83569b3a3c3e13f10fb078"}
Jul 16 23:13:56.139348 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.139296    2579 scope.go:117] "RemoveContainer" containerID="7d20f4dd70c67ed3b15d705f8795c39c4db2cbb78fd7fc50a329989b8c1a29ce"
Jul 16 23:13:56.159185 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.159161    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:56.165134 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.165114    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-75e8dc20-hskml"]
Jul 16 23:13:56.393692 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.393638    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrbck_e33299f5-3377-4a51-bdb2-478c0e3d6a7d/kube-rbac-proxy/0.log"
Jul 16 23:13:56.413202 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.413187    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrbck_e33299f5-3377-4a51-bdb2-478c0e3d6a7d/exporter/0.log"
Jul 16 23:13:56.433329 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.433306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrbck_e33299f5-3377-4a51-bdb2-478c0e3d6a7d/extractor/0.log"
Jul 16 23:13:56.460815 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:56.460794    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a39bd58-e1c2-4331-969c-903214d963a7" path="/var/lib/kubelet/pods/9a39bd58-e1c2-4331-969c-903214d963a7/volumes"
Jul 16 23:13:58.130988 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:58.130963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wvjp6/perf-node-gather-daemonset-wnbr7"
Jul 16 23:13:58.707169 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:58.707143    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-28m6q_1a53e0ad-e5b4-400c-80e7-9b0b416a23c1/manager/0.log"
Jul 16 23:13:59.259910 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:59.259882    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj_4927ca4e-c54d-47a0-9c70-30d1be286cfc/manager/0.log"
Jul 16 23:13:59.280332 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:13:59.280309    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cb9b8d7c8-n8spj_4927ca4e-c54d-47a0-9c70-30d1be286cfc/copy-manifests/0.log"
Jul 16 23:14:00.706548 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:00.706503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-67c6655946-mcr9w_9257138b-ca56-4040-8c75-b8f4d8b9187b/manager/0.log"
Jul 16 23:14:00.739970 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:00.739946    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-z7x6d_2d31da4d-b2c2-43c1-acf8-94c716a5a308/openshift-lws-operator/0.log"
Jul 16 23:14:01.359143 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:01.359118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b969c6b8c-kdw76_1d9a09f9-a705-4ca7-b9ab-002e598b7db3/kube-auth-proxy/0.log"
Jul 16 23:14:01.465208 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:01.465179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-7jm7k_765eb737-62ad-408f-bdc5-fa9dd064c524/payload-processing/0.log"
Jul 16 23:14:02.833808 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:02.833780    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b969c6b8c-kdw76_1d9a09f9-a705-4ca7-b9ab-002e598b7db3/kube-auth-proxy/0.log"
Jul 16 23:14:02.953641 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:02.953608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-7jm7k_765eb737-62ad-408f-bdc5-fa9dd064c524/payload-processing/0.log"
Jul 16 23:14:04.457124 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:04.457090    2579 scope.go:117] "RemoveContainer" containerID="dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3"
Jul 16 23:14:05.182026 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:05.181997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" event={"ID":"98724f2f-4f90-426d-a8d4-f531159c4d56","Type":"ContainerStarted","Data":"451b8be0c326ba37c4cb6a8748ff129e6624b8c8448996f1aeb34873069385da"}
Jul 16 23:14:05.199662 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:05.199613    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" podStartSLOduration=16.199598395 podStartE2EDuration="16.199598395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 23:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 23:14:05.197898089 +0000 UTC m=+3407.266221146" watchObservedRunningTime="2026-07-16 23:14:05.199598395 +0000 UTC m=+3407.267921449"
Jul 16 23:14:06.187778 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:06.187749    2579 generic.go:356] "Generic (PLEG): container finished" podID="98724f2f-4f90-426d-a8d4-f531159c4d56" containerID="451b8be0c326ba37c4cb6a8748ff129e6624b8c8448996f1aeb34873069385da" exitCode=6
Jul 16 23:14:06.188195 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:06.187822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" event={"ID":"98724f2f-4f90-426d-a8d4-f531159c4d56","Type":"ContainerDied","Data":"451b8be0c326ba37c4cb6a8748ff129e6624b8c8448996f1aeb34873069385da"}
Jul 16 23:14:06.188195 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:06.187882    2579 scope.go:117] "RemoveContainer" containerID="dbc9c4e3741d9c7caeab1294ca324864e4bbcaa978fcba787f7eda36b31254b3"
Jul 16 23:14:06.188195 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:06.188107    2579 scope.go:117] "RemoveContainer" containerID="451b8be0c326ba37c4cb6a8748ff129e6624b8c8448996f1aeb34873069385da"
Jul 16 23:14:06.188395 ip-10-0-141-9 kubenswrapper[2579]: E0716 23:14:06.188325    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-234edc78-v2v7l_odh-ai-gateway-infra(98724f2f-4f90-426d-a8d4-f531159c4d56)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l" podUID="98724f2f-4f90-426d-a8d4-f531159c4d56"
Jul 16 23:14:06.208661 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:06.208636    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"]
Jul 16 23:14:07.329823 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.329798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-234edc78-v2v7l"
Jul 16 23:14:07.372166 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.372145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/kube-multus-additional-cni-plugins/0.log"
Jul 16 23:14:07.393169 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.393146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/egress-router-binary-copy/0.log"
Jul 16 23:14:07.414689 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.414667    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/cni-plugins/0.log"
Jul 16 23:14:07.418958 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.418942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca\") pod \"98724f2f-4f90-426d-a8d4-f531159c4d56\" (UID: \"98724f2f-4f90-426d-a8d4-f531159c4d56\") "
Jul 16 23:14:07.419276 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.419255    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98724f2f-4f90-426d-a8d4-f531159c4d56" (UID: "98724f2f-4f90-426d-a8d4-f531159c4d56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 23:14:07.435490 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.435475    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/bond-cni-plugin/0.log"
Jul 16 23:14:07.457033 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.456983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/routeoverride-cni/0.log"
Jul 16 23:14:07.480562 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.480546    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/whereabouts-cni-bincopy/0.log"
Jul 16 23:14:07.500362 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.500341    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nz844_6382ec4b-30a4-468b-acd3-e885bc88981e/whereabouts-cni/0.log"
Jul 16 23:14:07.519662 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.519647    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98724f2f-4f90-426d-a8d4-f531159c4d56-maas-api-service-ca\") on node \"ip-10-0-141-9.ec2.internal\" DevicePath \"\""
Jul 16 23:14:07.578866 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.578846    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dnzvd_de6c1ef5-1296-409d-8d4e-e0a180b14f1b/kube-multus/0.log"
Jul 16 23:14:07.653488 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.653467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-57ngb_53207566-66e5-4cca-909d-754cbcb13ee1/network-metrics-daemon/0.log"
Jul 16 23:14:07.672665 ip-10-0-141-9 kubenswrapper[2579]: I0716 23:14:07.672647    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-57ngb_53207566-66e5-4cca-909d-754cbcb13ee1/kube-rbac-proxy/0.log"