Jul 19 07:42:43.371957 ip-10-0-129-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 19 07:42:44.225211 ip-10-0-129-91 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 07:42:44.225211 ip-10-0-129-91 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 19 07:42:44.225211 ip-10-0-129-91 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 07:42:44.225211 ip-10-0-129-91 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 19 07:42:44.225211 ip-10-0-129-91 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 19 07:42:44.226665 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.226549    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 19 07:42:44.233456 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233438    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233459    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233464    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233467    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233470    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233473    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233477    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233481    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233484    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233487    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233490    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233493    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233496    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233499    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233502    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.233500 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233506    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233511    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233514    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233519    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233522    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233525    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233529    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233532    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233535    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233538    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233542    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233546    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233549    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233552    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233554    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233557    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233560    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233563    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233566    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.233885 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233570    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233573    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233576    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233579    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233581    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233585    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233588    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233590    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233593    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233596    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233599    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233601    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233604    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233607    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233610    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233614    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233617    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233620    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233623    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.234403 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233625    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233629    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233632    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233635    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233637    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233640    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233643    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233646    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233648    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233651    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233654    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233656    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233659    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233662    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233665    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233668    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233670    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233673    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233676    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233679    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.234914 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233683    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233688    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233691    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233694    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233697    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233700    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233703    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233707    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233710    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233712    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233715    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233718    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233721    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233723    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233726    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233729    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233731    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233734    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233737    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233739    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.236940 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233742    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233745    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233748    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233750    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233753    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.233756    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234514    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234522    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234525    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234528    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234531    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234534    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234553    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234558    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234562    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234565    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234570    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234573    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234577    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234581    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.237863 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234584    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234587    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234589    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234593    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234595    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234598    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234601    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234604    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234607    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234610    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234612    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234615    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234618    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234621    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234624    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234626    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234629    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234631    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234634    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234637    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.238867 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234640    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234643    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234646    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234649    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234652    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234655    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234657    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234660    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234663    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234665    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234668    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234671    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234674    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234676    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234680    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234682    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234685    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234688    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234690    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234693    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.239562 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234696    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234699    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234703    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234707    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234710    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234713    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234716    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234719    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234722    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234724    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234728    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234730    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234735    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234739    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234743    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234746    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234749    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234752    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234755    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.240349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234757    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234761    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234764    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234766    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234769    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234790    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234793    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234798    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234803    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234806    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234809    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234812    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234815    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234818    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234821    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234824    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234826    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234829    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234832    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.241019 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234835    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234838    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234841    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234843    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234846    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234849    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.234851    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236767    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236783    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236789    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236794    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236799    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236805    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236810    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236815    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236820    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236824    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236829    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236834    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236840    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.241509 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236844    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236849    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236853    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236858    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236863    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236868    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236873    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236877    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236882    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236886    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236893    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236900    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236905    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236910    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236915    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236920    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236925    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236929    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236934    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.242054 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236943    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236948    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236955    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236960    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236965    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236970    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236974    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236979    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236983    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236988    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236992    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.236997    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237002    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237006    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237011    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237018    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237022    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237034    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237039    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.242652 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237044    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237048    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237053    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237057    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237062    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237066    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237071    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237076    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237080    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237084    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237089    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237094    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237098    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237102    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237107    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237112    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237116    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237121    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237125    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.243220 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237130    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237135    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237139    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237151    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237156    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237160    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237165    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237170    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237174    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237179    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237183    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237190    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237195    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237200    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237205    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237209    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237216    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237221    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237225    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237230    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.243708 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237237    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237243    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237248    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237253    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237258    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237262    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237267    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237273    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.237277    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237392    2559 flags.go:64] FLAG: --address="0.0.0.0"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237404    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237415    2559 flags.go:64] FLAG: --anonymous-auth="true"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237423    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237429    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237435    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237443    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237450    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237455    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237460    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237466    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237472    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237477    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 19 07:42:44.244240 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237483    2559 flags.go:64] FLAG: --cgroup-root=""
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237490    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237495    2559 flags.go:64] FLAG: --client-ca-file=""
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237501    2559 flags.go:64] FLAG: --cloud-provider="external"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237506    2559 flags.go:64] FLAG: --cluster-dns="[]"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237513    2559 flags.go:64] FLAG: --cluster-domain=""
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237518    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237524    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237529    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237535    2559 flags.go:64] FLAG: --container-log-max-files="5"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237541    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237547    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237552    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237557    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237563    2559 flags.go:64] FLAG: --contention-profiling="false"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237568    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237573    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237581    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237586    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237593    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237598    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237604    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237609    2559 flags.go:64] FLAG: --enable-load-reader="false"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237614    2559 flags.go:64] FLAG: --enable-server="true"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237618    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 19 07:42:44.244841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237627    2559 flags.go:64] FLAG: --event-burst="100"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237632    2559 flags.go:64] FLAG: --event-qps="50"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237637    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237643    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237648    2559 flags.go:64] FLAG: --eviction-hard=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237654    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237659    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237664    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237674    2559 flags.go:64] FLAG: --eviction-soft=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237680    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237698    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237704    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237709    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237715    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237719    2559 flags.go:64] FLAG: --fail-swap-on="true"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237724    2559 flags.go:64] FLAG: --feature-gates=""
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237737    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237743    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237748    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237753    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237758    2559 flags.go:64] FLAG: --healthz-port="10248"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237764    2559 flags.go:64] FLAG: --help="false"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237769    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237774    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 19 07:42:44.245531 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237794    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237800    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237806    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237812    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237817    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237822    2559 flags.go:64] FLAG: --image-service-endpoint=""
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237826    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237853    2559 flags.go:64] FLAG: --kube-api-burst="100"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237860    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237866    2559 flags.go:64] FLAG: --kube-api-qps="50"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237871    2559 flags.go:64] FLAG: --kube-reserved=""
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237876    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237881    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237886    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237891    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237896    2559 flags.go:64] FLAG: --lock-file=""
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237903    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237908    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237914    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237924    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237930    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237935    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237940    2559 flags.go:64] FLAG: --logging-format="text"
Jul 19 07:42:44.246246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237945    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237951    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237956    2559 flags.go:64] FLAG: --manifest-url=""
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237961    2559 flags.go:64] FLAG: --manifest-url-header=""
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237969    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237974    2559 flags.go:64] FLAG: --max-open-files="1000000"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237981    2559 flags.go:64] FLAG: --max-pods="110"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237986    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237991    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.237997    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238002    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238007    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238012    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238017    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238028    2559 flags.go:64] FLAG: --node-status-max-images="50"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238033    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238038    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238043    2559 flags.go:64] FLAG: --pod-cidr=""
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238048    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238058    2559 flags.go:64] FLAG: --pod-manifest-path=""
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238063    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238068    2559 flags.go:64] FLAG: --pods-per-core="0"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238073    2559 flags.go:64] FLAG: --port="10250"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238078    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 19 07:42:44.246863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238083    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e2fa9242cd2fa88b"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238090    2559 flags.go:64] FLAG: --qos-reserved=""
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238096    2559 flags.go:64] FLAG: --read-only-port="10255"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238101    2559 flags.go:64] FLAG: --register-node="true"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238105    2559 flags.go:64] FLAG: --register-with-taints=""
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238112    2559 flags.go:64] FLAG: --registry-burst="10"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238117    2559 flags.go:64] FLAG: --registry-qps="5"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238122    2559 flags.go:64] FLAG: --reserved-cpus=""
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238128    2559 flags.go:64] FLAG: --reserved-memory=""
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238134    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238139    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238145    2559 flags.go:64] FLAG: --rotate-certificates="false"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238150    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238155    2559 flags.go:64] FLAG: --runonce="false"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238160    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238165    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238170    2559 flags.go:64] FLAG: --seccomp-default="false"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238175    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238181    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238186    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238192    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238197    2559 flags.go:64] FLAG: --storage-driver-password="root"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238202    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238207    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238212    2559 flags.go:64] FLAG: --storage-driver-user="root"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238216    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 19 07:42:44.247485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238222    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238227    2559 flags.go:64] FLAG: --system-cgroups=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238232    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238242    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238247    2559 flags.go:64] FLAG: --tls-cert-file=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238251    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238259    2559 flags.go:64] FLAG: --tls-min-version=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238266    2559 flags.go:64] FLAG: --tls-private-key-file=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238271    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238276    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238281    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238287    2559 flags.go:64] FLAG: --v="2"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238294    2559 flags.go:64] FLAG: --version="false"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238302    2559 flags.go:64] FLAG: --vmodule=""
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238309    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.238315    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238513    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238520    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238525    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238530    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238535    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238540    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238544    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.248181 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238549    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238554    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238559    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238564    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238571    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238577    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238582    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238587    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238592    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238598    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238603    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238607    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238612    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238616    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238621    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238625    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238634    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238642    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238663    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.248846 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238668    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238673    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238677    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238682    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238687    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238691    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238695    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238713    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238717    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238722    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238727    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238731    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238735    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238740    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238745    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238750    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238757    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238763    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238768    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238773    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.249349 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238779    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238784    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238789    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238794    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238798    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238803    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238807    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238812    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238816    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238823    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238828    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238833    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238837    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238842    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238847    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238851    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238856    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238862    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238867    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238871    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.249894 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238876    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238880    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238884    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238889    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238893    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238898    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238907    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238911    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238916    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238921    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238926    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238930    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238935    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238939    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238944    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238948    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238953    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238957    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238961    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238966    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.250451 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238970    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238975    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238980    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238984    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238989    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238993    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.238998    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239003    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239007    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239012    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239017    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239022    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.239026    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.239040    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.247112    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 19 07:42:44.250974 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.247127    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.247948    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.247957    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248018    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248023    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248027    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248031    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248034    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248037    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248040    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248044    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248047    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248050    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248053    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248056    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248059    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248062    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248065    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248068    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.251382 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248072    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248076    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248079    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248082    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248085    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248088    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248091    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248094    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248096    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248099    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248102    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248104    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248107    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248112    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248115    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248118    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248121    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248124    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248127    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248130    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.251888 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248132    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248135    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248138    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248141    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248143    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248146    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248149    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248152    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248154    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248157    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248160    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248163    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248165    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248168    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248170    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248173    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248176    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248178    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248181    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248184    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.252416 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248187    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248190    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248193    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248196    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248199    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248203    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248206    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248209    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248212    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248215    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248218    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248220    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248223    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248226    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248229    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248231    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248234    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248237    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248239    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248242    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.252949 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248245    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248248    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248250    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248253    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248256    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248259    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248261    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248264    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248267    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248270    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248272    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248275    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248279    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248283    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248286    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248290    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248294    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248298    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248301    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.253455 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248305    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248308    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248311    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248313    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.248319    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248440    2559 feature_gate.go:328] unrecognized feature gate: Example2
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248445    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248448    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248451    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248454    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248457    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248461    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248464    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248467    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248469    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 19 07:42:44.254047 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248472    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248475    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248477    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248480    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248483    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248486    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248488    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248491    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248494    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248496    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248499    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248502    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248504    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248507    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248510    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248513    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248516    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248519    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248522    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248525    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 19 07:42:44.254442 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248528    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248531    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248534    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248536    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248539    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248542    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248544    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248547    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248550    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248552    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248555    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248558    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248561    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248564    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248566    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248569    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248572    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248574    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248577    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 19 07:42:44.254972 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248579    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248582    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248585    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248587    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248590    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248593    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248596    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248598    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248601    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248605    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248608    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248610    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248619    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248622    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248625    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248628    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248631    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248635    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248638    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248642    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 19 07:42:44.255461 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248645    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248647    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248650    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248653    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248656    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248659    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248662    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248664    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248667    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248670    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248673    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248675    2559 feature_gate.go:328] unrecognized feature gate: Example
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248678    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248681    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248683    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248686    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248689    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248692    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248695    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248698    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 19 07:42:44.256002 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248701    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248706    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248709    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248712    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248715    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248719    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248722    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248724    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248728    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:44.248732    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.248737    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.248869    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.251308    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 19 07:42:44.256516 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.252675    2559 server.go:1019] "Starting client certificate rotation"
Jul 19 07:42:44.256880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.252810    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 19 07:42:44.256880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.252845    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 19 07:42:44.256880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.255448    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 19 07:42:44.258941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.258923    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 19 07:42:44.272269 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.272252    2559 log.go:25] "Validated CRI v1 runtime API"
Jul 19 07:42:44.274422 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.274409    2559 log.go:25] "Validated CRI v1 image API"
Jul 19 07:42:44.275599 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.275578    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 19 07:42:44.278387 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.278367    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8bbc82fa-0113-4b10-ad49-9f6ac2f7ffe0:/dev/nvme0n1p4 c358b2a4-dab9-47e7-82ac-def520dfd919:/dev/nvme0n1p3]
Jul 19 07:42:44.278456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.278386    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 19 07:42:44.282058 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.282031    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 19 07:42:44.285607 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.285490    2559 manager.go:216] Machine: {Timestamp:2026-07-19 07:42:44.284173234 +0000 UTC m=+0.793771127 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087067 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27b60dcdc6e18cd649f97b574e426c SystemUUID:ec27b60d-cdc6-e18c-d649-f97b574e426c BootID:0afb0e41-016e-41a5-a5fe-9fec19f11419 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fe:d8:93:20:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fe:d8:93:20:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:8c:36:28:22:f1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 19 07:42:44.285607 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.285601    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 19 07:42:44.285717 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.285680    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 19 07:42:44.289300 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.289273    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 19 07:42:44.289441 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.289303    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 19 07:42:44.290060 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.290048    2559 topology_manager.go:138] "Creating topology manager with none policy"
Jul 19 07:42:44.290101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.290062    2559 container_manager_linux.go:309] "Creating device plugin manager"
Jul 19 07:42:44.290101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.290070    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 19 07:42:44.290874 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.290862    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 19 07:42:44.292101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.292091    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 19 07:42:44.296036 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296024    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 19 07:42:44.296164 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296154    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 19 07:42:44.296201 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296187    2559 kubelet.go:486] "Attempting to sync node with API server"
Jul 19 07:42:44.296201 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296198    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 19 07:42:44.296257 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296217    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 19 07:42:44.296257 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296230    2559 kubelet.go:398] "Adding apiserver pod source"
Jul 19 07:42:44.296257 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296240    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 19 07:42:44.296664 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296651    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 19 07:42:44.296726 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.296675    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 19 07:42:44.298923 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.298906    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 19 07:42:44.300809 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.300795    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 19 07:42:44.300874 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.300828    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 19 07:42:44.302005 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.301991    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302008    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302015    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302021    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302030    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302036    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302050    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302056    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302063    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302069    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 19 07:42:44.302082 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302085    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 19 07:42:44.302353 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302091    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 19 07:42:44.302941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302931    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 19 07:42:44.302941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.302941    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 19 07:42:44.305410 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.305391    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-86k7h"
Jul 19 07:42:44.305838 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.305815    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 19 07:42:44.305879 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.305843    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 19 07:42:44.306112 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.306078    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 19 07:42:44.306590 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.306577    2559 server.go:1268] "Started kubelet"
Jul 19 07:42:44.306700 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.306654    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 19 07:42:44.306774 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.306677    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 19 07:42:44.306774 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.306759    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 19 07:42:44.307580 ip-10-0-129-91 systemd[1]: Started Kubernetes Kubelet.
Jul 19 07:42:44.308358 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.308337    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 19 07:42:44.308967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.308952    2559 server.go:310] "Adding debug handlers to kubelet server"
Jul 19 07:42:44.311414 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.311398    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-86k7h"
Jul 19 07:42:44.313352 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.313327    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 19 07:42:44.313818 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.312884    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-91.ec2.internal.18c3a10f4404192a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-91.ec2.internal,UID:ip-10-0-129-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-91.ec2.internal,},FirstTimestamp:2026-07-19 07:42:44.306549034 +0000 UTC m=+0.816146927,LastTimestamp:2026-07-19 07:42:44.306549034 +0000 UTC m=+0.816146927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-91.ec2.internal,}"
Jul 19 07:42:44.316739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.316719    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 19 07:42:44.316855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.316741    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 19 07:42:44.316855 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.316805    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.317454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317434    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 19 07:42:44.317454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317456    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 19 07:42:44.317617 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317434    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 19 07:42:44.317673 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317636    2559 factory.go:55] Registering systemd factory
Jul 19 07:42:44.317673 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317651    2559 factory.go:223] Registration of the systemd container factory successfully
Jul 19 07:42:44.317796 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317677    2559 reconstruct.go:97] "Volume reconstruction finished"
Jul 19 07:42:44.317796 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317685    2559 reconciler.go:29] "Reconciler: start to sync state"
Jul 19 07:42:44.317978 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317960    2559 factory.go:153] Registering CRI-O factory
Jul 19 07:42:44.317978 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.317979    2559 factory.go:223] Registration of the crio container factory successfully
Jul 19 07:42:44.318205 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.318032    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 19 07:42:44.318205 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.318059    2559 factory.go:103] Registering Raw factory
Jul 19 07:42:44.318205 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.318074    2559 manager.go:1196] Started watching for new ooms in manager
Jul 19 07:42:44.318476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.318459    2559 manager.go:349] Starting recovery of all containers
Jul 19 07:42:44.321306 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.321285    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:44.326656 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.326628    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-91.ec2.internal\" not found" node="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.331158 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.331141    2559 manager.go:354] Recovery completed
Jul 19 07:42:44.335636 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.335619    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.337475 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.337459    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.337570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.337493    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.337570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.337509    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.338088 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338076    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.338145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338079    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 19 07:42:44.338145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338107    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 19 07:42:44.338145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338124    2559 state_mem.go:36] "Initialized new in-memory state store"
Jul 19 07:42:44.338895 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338880    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.338969 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338911    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.338969 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.338924    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.339272 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.339261    2559 policy_none.go:49] "None policy: Start"
Jul 19 07:42:44.339309 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.339278    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 19 07:42:44.339309 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.339289    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 19 07:42:44.340262 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.340252    2559 policy_none.go:47] "Start"
Jul 19 07:42:44.373407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373280    2559 manager.go:337] "Starting Device Plugin manager"
Jul 19 07:42:44.373524 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.373447    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 19 07:42:44.373524 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373465    2559 server.go:87] "Starting device plugin registration server"
Jul 19 07:42:44.373736 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373712    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 19 07:42:44.373800 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373728    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 19 07:42:44.373851 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373797    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 19 07:42:44.373911 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373881    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 19 07:42:44.373911 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.373891    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 19 07:42:44.374478 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.374454    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 19 07:42:44.374558 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.374513    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.452966 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.452933    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 19 07:42:44.454279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.454131    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 19 07:42:44.454279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.454174    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 19 07:42:44.454279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.454199    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 19 07:42:44.454279 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.454236    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 19 07:42:44.456519 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.456498    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:44.474621 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.474602    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.476140 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.476089    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.476140 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.476123    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.476140 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.476134    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.476292 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.476158    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.488138 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.488123    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.488192 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.488147    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-91.ec2.internal\": node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.503000 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.502976    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.556499 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.556471    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"]
Jul 19 07:42:44.556581 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.556544    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.558236 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.558220    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.558320 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.558248    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.558320 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.558259    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.559408 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.559395    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.559544 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.559529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.559583 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.559565    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.560203 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560185    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.560277 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560217    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.560277 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560229    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.560277 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560193    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.560426 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560294    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.560426 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.560310    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.561644 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.561627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.561728 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.561658    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 19 07:42:44.562912 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.562897    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 19 07:42:44.562999 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.562926    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 19 07:42:44.562999 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.562941    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeHasSufficientPID"
Jul 19 07:42:44.579348 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.579333    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-91.ec2.internal\" not found" node="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.583655 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.583641    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-91.ec2.internal\" not found" node="ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.603186 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.603165    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.703779 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.703737    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.721393 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.721376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.721455 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.721400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.721455 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.721421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/338760c60ab411de52b406a82ca30247-config\") pod \"kube-apiserver-proxy-ip-10-0-129-91.ec2.internal\" (UID: \"338760c60ab411de52b406a82ca30247\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.804030 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.803944    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:44.822224 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/338760c60ab411de52b406a82ca30247-config\") pod \"kube-apiserver-proxy-ip-10-0-129-91.ec2.internal\" (UID: \"338760c60ab411de52b406a82ca30247\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.822315 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.822315 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.822315 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.822480 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5459af162c0ff1eb3c23b44d0dcf199a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal\" (UID: \"5459af162c0ff1eb3c23b44d0dcf199a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.822480 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.822309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/338760c60ab411de52b406a82ca30247-config\") pod \"kube-apiserver-proxy-ip-10-0-129-91.ec2.internal\" (UID: \"338760c60ab411de52b406a82ca30247\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.882314 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.882275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.888528 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:44.888511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:44.904727 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:44.904690    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.006317 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.006277    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.112912 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.112875    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.214730 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.214680    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.252991 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.252971    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 19 07:42:45.253406 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.253109    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 19 07:42:45.253406 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.253141    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 19 07:42:45.315430 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.315406    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.316792 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.316758    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 07:37:44 +0000 UTC" deadline="2028-03-03 01:50:19.467009593 +0000 UTC"
Jul 19 07:42:45.316847 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.316789    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14226h7m34.150225517s"
Jul 19 07:42:45.415984 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.415921    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.429562 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.429531    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:45.451321 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:45.451278    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5459af162c0ff1eb3c23b44d0dcf199a.slice/crio-8a980977e8ca55213cea58232e1b60a28b90442c3cfd4c830641dafd3fad01d0 WatchSource:0}: Error finding container 8a980977e8ca55213cea58232e1b60a28b90442c3cfd4c830641dafd3fad01d0: Status 404 returned error can't find the container with id 8a980977e8ca55213cea58232e1b60a28b90442c3cfd4c830641dafd3fad01d0
Jul 19 07:42:45.451864 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:45.451846    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338760c60ab411de52b406a82ca30247.slice/crio-91ae1c36d645564076f3643e59149a276a7852db27a3d49c4035e5a9756aa8bd WatchSource:0}: Error finding container 91ae1c36d645564076f3643e59149a276a7852db27a3d49c4035e5a9756aa8bd: Status 404 returned error can't find the container with id 91ae1c36d645564076f3643e59149a276a7852db27a3d49c4035e5a9756aa8bd
Jul 19 07:42:45.457682 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.457662    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 07:42:45.457768 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.457669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal" event={"ID":"338760c60ab411de52b406a82ca30247","Type":"ContainerStarted","Data":"91ae1c36d645564076f3643e59149a276a7852db27a3d49c4035e5a9756aa8bd"}
Jul 19 07:42:45.458880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.458853    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal" event={"ID":"5459af162c0ff1eb3c23b44d0dcf199a","Type":"ContainerStarted","Data":"8a980977e8ca55213cea58232e1b60a28b90442c3cfd4c830641dafd3fad01d0"}
Jul 19 07:42:45.519768 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.519739    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.620155 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.620128    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.720630 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.720558    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.820920 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:45.820899    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-91.ec2.internal\" not found"
Jul 19 07:42:45.883780 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.883762    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:45.917856 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.917824    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:45.930009 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.929986    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 19 07:42:45.931181 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.931159    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal"
Jul 19 07:42:45.937146 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:45.937126    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 19 07:42:46.093545 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.093511    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:46.201097 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.200891    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 19 07:42:46.298302 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.298208    2559 apiserver.go:52] "Watching apiserver"
Jul 19 07:42:46.305406 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.305370    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 19 07:42:46.306157 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.305867    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-jrn9b","openshift-network-diagnostics/network-check-target-qn6rj","kube-system/konnectivity-agent-vx5wp","kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal","openshift-cluster-node-tuning-operator/tuned-b4njh","openshift-image-registry/node-ca-m7wmg","openshift-multus/network-metrics-daemon-xnkds","openshift-network-operator/iptables-alerter-552d8","openshift-ovn-kubernetes/ovnkube-node-qbtsm","kube-system/global-pull-secret-syncer-p28rd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc","openshift-dns/node-resolver-g2dcs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal","openshift-multus/multus-8z8rk"]
Jul 19 07:42:46.307533 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.307515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.309228 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.309205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:46.309328 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.309284    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:46.309913 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.309891    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.310005 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.309911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 19 07:42:46.310071 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.310015    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zkqft\""
Jul 19 07:42:46.310123 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.310093    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.310422 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.310403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.311611 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.311592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.312803 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.312780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.312901 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.312845    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:46.313308 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 19 07:42:46.313308 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m46r8\""
Jul 19 07:42:46.313308 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313056    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 19 07:42:46.313646 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313631    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.313996 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313974    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.314140 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.313984    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bnq79\""
Jul 19 07:42:46.314442 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.314414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.315662 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.315641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.315777 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.315707    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:46.316732 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.316697    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 19 07:42:46.316969 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.316945    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 19 07:42:46.317067 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.316982    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.317197 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.317181    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.317513 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.317282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7b449\""
Jul 19 07:42:46.317513 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.317369    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.319624 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.319107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.321045 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.320597    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkr9x\""
Jul 19 07:42:46.321045 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.320977    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.321045 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.321006    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.321242 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.321174    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.322683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.321566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 19 07:42:46.322683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.322147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.322683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.322392    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 19 07:42:46.323418 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.323398    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.323553 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.323533    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 19 07:42:46.323814 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.323795    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.323901 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.323883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 19 07:42:46.324009 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.323992    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v6bd7\""
Jul 19 07:42:46.324088 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324010    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 19 07:42:46.324145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324096    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 19 07:42:46.324203 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.324549 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324526    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 19 07:42:46.324634 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324623    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 19 07:42:46.324793 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324766    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.324921 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.324904    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w5b7k\""
Jul 19 07:42:46.325361 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.325339    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.327204 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.327184    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 19 07:42:46.327543 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.327524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-j6g72\""
Jul 19 07:42:46.327831 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.327810    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 19 07:42:46.327956 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.327857    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5qkv5\""
Jul 19 07:42:46.327956 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.327858    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 19 07:42:46.329842 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.329823    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-slash\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.329941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.329875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.329941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.329921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktmk\" (UniqueName: \"kubernetes.io/projected/3e647a7e-bae7-4e90-97ee-a174eb876813-kube-api-access-pktmk\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.329948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.330062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.329975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.330062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd56g\" (UniqueName: \"kubernetes.io/projected/76520261-12a8-4ef9-b096-6a879578622b-kube-api-access-gd56g\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.330062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb3a82c-68ca-4f01-9443-7467016acdb7-host-slash\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330067    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-etc-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330133    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-script-lib\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330192    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6h4x\" (UniqueName: \"kubernetes.io/projected/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-kube-api-access-q6h4x\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.330255 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330240    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0eb070-3b10-43df-950b-f9ac992e9049-tmp-dir\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-socket-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-dbus\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhdx\" (UniqueName: \"kubernetes.io/projected/0bb3a82c-68ca-4f01-9443-7467016acdb7-kube-api-access-fjhdx\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmm7x\" (UniqueName: \"kubernetes.io/projected/0a0eb070-3b10-43df-950b-f9ac992e9049-kube-api-access-pmm7x\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b47f2f7d-461c-40e1-9919-439c2dc8f02d-host\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-netns\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-ovn\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330405    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-node-log\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-etc-selinux\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-kubelet-config\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.330525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-registration-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-sys-fs\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-kubelet\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-systemd-units\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-systemd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-var-lib-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-bin\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330704    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d332895-40aa-49e6-a1c2-33879f175472-agent-certs\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330728    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-device-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-log-socket\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-config\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb3a82c-68ca-4f01-9443-7467016acdb7-iptables-alerter-script\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-netd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-env-overrides\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330947    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e647a7e-bae7-4e90-97ee-a174eb876813-ovn-node-metrics-cert\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.331068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b47f2f7d-461c-40e1-9919-439c2dc8f02d-serviceca\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.331712 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.330986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lks7\" (UniqueName: \"kubernetes.io/projected/b47f2f7d-461c-40e1-9919-439c2dc8f02d-kube-api-access-4lks7\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.331712 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.331006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d332895-40aa-49e6-a1c2-33879f175472-konnectivity-ca\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.331712 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.331038    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0eb070-3b10-43df-950b-f9ac992e9049-hosts-file\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.334462 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.334275    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 19 07:42:46.351373 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.351335    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b4x5t"
Jul 19 07:42:46.364386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.364366    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b4x5t"
Jul 19 07:42:46.419455 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.419422    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 19 07:42:46.431654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-systemd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.431790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431662    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-var-lib-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.431790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-multus-certs\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.431790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-systemd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.431790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-os-release\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.431790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431791    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-var-lib-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d332895-40aa-49e6-a1c2-33879f175472-agent-certs\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-device-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-socket-dir-parent\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.431952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-system-cni-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-device-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.432073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-var-lib-kubelet\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-tuned\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-env-overrides\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e647a7e-bae7-4e90-97ee-a174eb876813-ovn-node-metrics-cert\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432284    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-bin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432367    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysconfig\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.432403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lks7\" (UniqueName: \"kubernetes.io/projected/b47f2f7d-461c-40e1-9919-439c2dc8f02d-kube-api-access-4lks7\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432421    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0eb070-3b10-43df-950b-f9ac992e9049-hosts-file\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-slash\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-hostroot\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d842\" (UniqueName: \"kubernetes.io/projected/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-kube-api-access-9d842\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.432723 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0eb070-3b10-43df-950b-f9ac992e9049-hosts-file\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432902    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-slash\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-script-lib\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-env-overrides\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.432997 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.432962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-netns\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.433695 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-daemon-config\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.433807 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.433881 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6h4x\" (UniqueName: \"kubernetes.io/projected/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-kube-api-access-q6h4x\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.433881 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-socket-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.434001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhdx\" (UniqueName: \"kubernetes.io/projected/0bb3a82c-68ca-4f01-9443-7467016acdb7-kube-api-access-fjhdx\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.434001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-script-lib\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.434001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-node-log\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.434001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.433982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cni-binary-copy\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434214 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434012    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-kubelet\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-socket-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.434325 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-node-log\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.434471 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-etc-kubernetes\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434542 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.434604 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434542    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-sys\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.434604 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-kubelet-config\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.434704 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434756 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-os-release\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434835 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-conf-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.434835 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434798    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-registration-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.434835 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-sys-fs\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.434990 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-bin\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.434990 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-tmp\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.434990 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-log-socket\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.435194 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.434989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-config\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.435194 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj44c\" (UniqueName: \"kubernetes.io/projected/5daba15c-5b35-477d-9154-ad2c0f2a108a-kube-api-access-lj44c\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.435194 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-kubernetes\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435194 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-systemd\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435194 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-lib-modules\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435432 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb3a82c-68ca-4f01-9443-7467016acdb7-iptables-alerter-script\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.435432 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-netd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.435432 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-system-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.435432 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435381    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cnibin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.435432 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435413    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-conf\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b47f2f7d-461c-40e1-9919-439c2dc8f02d-serviceca\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d332895-40aa-49e6-a1c2-33879f175472-konnectivity-ca\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktmk\" (UniqueName: \"kubernetes.io/projected/3e647a7e-bae7-4e90-97ee-a174eb876813-kube-api-access-pktmk\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.435654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-host\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.435952 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-kubelet-config\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.435952 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.435789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.436533 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.436533 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436507    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bb3a82c-68ca-4f01-9443-7467016acdb7-iptables-alerter-script\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.436711 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-registration-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.436711 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-sys-fs\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.436711 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436722    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-bin\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.436899 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.436804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-log-socket\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.437285 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437262    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e647a7e-bae7-4e90-97ee-a174eb876813-ovnkube-config\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.437363 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.437363 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b47f2f7d-461c-40e1-9919-439c2dc8f02d-serviceca\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.437479 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.437479 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.437457    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:46.437668 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd56g\" (UniqueName: \"kubernetes.io/projected/76520261-12a8-4ef9-b096-6a879578622b-kube-api-access-gd56g\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.437668 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.437663    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:46.937641625 +0000 UTC m=+3.447239511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:46.437814 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.437702    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb3a82c-68ca-4f01-9443-7467016acdb7-host-slash\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.438079 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438056    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-cni-netd\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438159 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-etc-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438230 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438308 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d332895-40aa-49e6-a1c2-33879f175472-konnectivity-ca\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.438359 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-k8s-cni-cncf-io\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.438359 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-ovn-kubernetes\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438463 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0eb070-3b10-43df-950b-f9ac992e9049-tmp-dir\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.438573 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438484    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bb3a82c-68ca-4f01-9443-7467016acdb7-host-slash\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.438632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438571    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-etc-openvswitch\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-dbus\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.438734 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438674    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-netns\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-ovn\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.438889 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmm7x\" (UniqueName: \"kubernetes.io/projected/0a0eb070-3b10-43df-950b-f9ac992e9049-kube-api-access-pmm7x\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.438889 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b47f2f7d-461c-40e1-9919-439c2dc8f02d-host\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.438995 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438890    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-multus\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.438995 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.438988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-cnibin\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.439104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d332895-40aa-49e6-a1c2-33879f175472-agent-certs\") pod \"konnectivity-agent-vx5wp\" (UID: \"1d332895-40aa-49e6-a1c2-33879f175472\") " pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.439104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-modprobe-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.439104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-run\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.439104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-run-netns\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-dbus\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-run-ovn\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-etc-selinux\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nj6g\" (UniqueName: \"kubernetes.io/projected/f392427d-b56a-4351-b5a2-47d8dac7c6a5-kube-api-access-9nj6g\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.439319 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.439638 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-kubelet\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439638 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-systemd-units\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439638 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-systemd-units\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439808 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e647a7e-bae7-4e90-97ee-a174eb876813-host-kubelet\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.439808 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0eb070-3b10-43df-950b-f9ac992e9049-tmp-dir\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.439914 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b47f2f7d-461c-40e1-9919-439c2dc8f02d-host\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.440167 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.439985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76520261-12a8-4ef9-b096-6a879578622b-etc-selinux\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.440167 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.440144    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:46.440357 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.440317    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:42:46.940201632 +0000 UTC m=+3.449799526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:46.441719 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.441665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e647a7e-bae7-4e90-97ee-a174eb876813-ovn-node-metrics-cert\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.443130 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.443103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lks7\" (UniqueName: \"kubernetes.io/projected/b47f2f7d-461c-40e1-9919-439c2dc8f02d-kube-api-access-4lks7\") pod \"node-ca-m7wmg\" (UID: \"b47f2f7d-461c-40e1-9919-439c2dc8f02d\") " pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.444732 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.444708    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6h4x\" (UniqueName: \"kubernetes.io/projected/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-kube-api-access-q6h4x\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.444835 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.444767    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhdx\" (UniqueName: \"kubernetes.io/projected/0bb3a82c-68ca-4f01-9443-7467016acdb7-kube-api-access-fjhdx\") pod \"iptables-alerter-552d8\" (UID: \"0bb3a82c-68ca-4f01-9443-7467016acdb7\") " pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.445514 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.445498    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:42:46.445609 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.445522    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:42:46.445609 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.445537    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:46.445609 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.445602    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:46.9455843 +0000 UTC m=+3.455182198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:46.447877 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.447848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktmk\" (UniqueName: \"kubernetes.io/projected/3e647a7e-bae7-4e90-97ee-a174eb876813-kube-api-access-pktmk\") pod \"ovnkube-node-qbtsm\" (UID: \"3e647a7e-bae7-4e90-97ee-a174eb876813\") " pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.447877 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.447862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd56g\" (UniqueName: \"kubernetes.io/projected/76520261-12a8-4ef9-b096-6a879578622b-kube-api-access-gd56g\") pod \"aws-ebs-csi-driver-node-msptc\" (UID: \"76520261-12a8-4ef9-b096-6a879578622b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.448318 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.448294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmm7x\" (UniqueName: \"kubernetes.io/projected/0a0eb070-3b10-43df-950b-f9ac992e9049-kube-api-access-pmm7x\") pod \"node-resolver-g2dcs\" (UID: \"0a0eb070-3b10-43df-950b-f9ac992e9049\") " pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.539739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-multus\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.539739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-cnibin\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-modprobe-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-run\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539851    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-cnibin\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-multus\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nj6g\" (UniqueName: \"kubernetes.io/projected/f392427d-b56a-4351-b5a2-47d8dac7c6a5-kube-api-access-9nj6g\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-multus-certs\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539931    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-run\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.539957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-os-release\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-multus-certs\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-socket-dir-parent\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.539997    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-modprobe-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-system-cni-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-os-release\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540089    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-system-cni-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-socket-dir-parent\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-var-lib-kubelet\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-tuned\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540212    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-bin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysconfig\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-var-lib-kubelet\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-hostroot\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-cni-bin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.540386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-hostroot\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysconfig\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540399    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d842\" (UniqueName: \"kubernetes.io/projected/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-kube-api-access-9d842\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-netns\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540456    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-daemon-config\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540512    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-netns\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cni-binary-copy\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-kubelet\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-etc-kubernetes\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5daba15c-5b35-477d-9154-ad2c0f2a108a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-var-lib-kubelet\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-sys\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-etc-kubernetes\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.541231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-sys\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-os-release\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-conf-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-tmp\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540926    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-conf-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-os-release\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj44c\" (UniqueName: \"kubernetes.io/projected/5daba15c-5b35-477d-9154-ad2c0f2a108a-kube-api-access-lj44c\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.540976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-kubernetes\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-systemd\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-lib-modules\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-system-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cnibin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541150    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-multus-daemon-config\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541198    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cni-binary-copy\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541211    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-conf\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.542151 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5daba15c-5b35-477d-9154-ad2c0f2a108a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-kubernetes\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-system-cni-dir\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-lib-modules\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541288    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-systemd\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-cnibin\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-host\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-d\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-sysctl-conf\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-host\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-k8s-cni-cncf-io\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.541458    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f392427d-b56a-4351-b5a2-47d8dac7c6a5-host-run-k8s-cni-cncf-io\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.543149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.542903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-etc-tuned\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.543616 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.543489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-tmp\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.548118 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.548090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nj6g\" (UniqueName: \"kubernetes.io/projected/f392427d-b56a-4351-b5a2-47d8dac7c6a5-kube-api-access-9nj6g\") pod \"multus-8z8rk\" (UID: \"f392427d-b56a-4351-b5a2-47d8dac7c6a5\") " pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.548376 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.548358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d842\" (UniqueName: \"kubernetes.io/projected/760ff2a6-7b0c-4d1d-919a-b7e4029ef71e-kube-api-access-9d842\") pod \"tuned-b4njh\" (UID: \"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e\") " pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.548623 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.548596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj44c\" (UniqueName: \"kubernetes.io/projected/5daba15c-5b35-477d-9154-ad2c0f2a108a-kube-api-access-lj44c\") pod \"multus-additional-cni-plugins-jrn9b\" (UID: \"5daba15c-5b35-477d-9154-ad2c0f2a108a\") " pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.621820 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.621726    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc"
Jul 19 07:42:46.630719 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.630694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:42:46.654760 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.654735    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jrn9b"
Jul 19 07:42:46.655808 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.655780    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2dcs"
Jul 19 07:42:46.663575 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.663552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7wmg"
Jul 19 07:42:46.671661 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.671634    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-552d8"
Jul 19 07:42:46.679578 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.679558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:42:46.686339 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.686322    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b4njh"
Jul 19 07:42:46.691922 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.691900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8z8rk"
Jul 19 07:42:46.945419 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.945336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:46.945574 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:46.945424    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:46.945574 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.945499    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:46.945574 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.945547    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:46.945699 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.945575    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:42:47.945556061 +0000 UTC m=+4.455153941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:46.945699 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:46.945603    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:47.945584503 +0000 UTC m=+4.455182387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:47.046564 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.046521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:47.046767 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.046693    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:42:47.046767 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.046740    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:42:47.046767 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.046755    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:47.046947 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.046822    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:48.046800897 +0000 UTC m=+4.556398791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:47.367482 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.367444    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:46 +0000 UTC" deadline="2028-01-13 13:56:14.125081064 +0000 UTC"
Jul 19 07:42:47.367482 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.367474    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13038h13m26.757610517s"
Jul 19 07:42:47.455002 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.454972    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:47.455177 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.455127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:47.460776 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.460565    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0eb070_3b10_43df_950b_f9ac992e9049.slice/crio-d425a9b08bd1835ed6431b53c33792775f526efd470873bf2ff01ea43b582e96 WatchSource:0}: Error finding container d425a9b08bd1835ed6431b53c33792775f526efd470873bf2ff01ea43b582e96: Status 404 returned error can't find the container with id d425a9b08bd1835ed6431b53c33792775f526efd470873bf2ff01ea43b582e96
Jul 19 07:42:47.461874 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.461840    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e647a7e_bae7_4e90_97ee_a174eb876813.slice/crio-c0d65790f06db7cb0dfd2a0ce7602218fdc6fe0ae34a8781723d3ed784359aa3 WatchSource:0}: Error finding container c0d65790f06db7cb0dfd2a0ce7602218fdc6fe0ae34a8781723d3ed784359aa3: Status 404 returned error can't find the container with id c0d65790f06db7cb0dfd2a0ce7602218fdc6fe0ae34a8781723d3ed784359aa3
Jul 19 07:42:47.465758 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.465733    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76520261_12a8_4ef9_b096_6a879578622b.slice/crio-b318d70176c31b7069eba0f91914740b940002181c9991afd48bd99163be6b32 WatchSource:0}: Error finding container b318d70176c31b7069eba0f91914740b940002181c9991afd48bd99163be6b32: Status 404 returned error can't find the container with id b318d70176c31b7069eba0f91914740b940002181c9991afd48bd99163be6b32
Jul 19 07:42:47.466824 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.466797    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d332895_40aa_49e6_a1c2_33879f175472.slice/crio-8d8f67dc938617ec74e2fd782e47b54c69dc64d36b78191bbb2dcf227af6117e WatchSource:0}: Error finding container 8d8f67dc938617ec74e2fd782e47b54c69dc64d36b78191bbb2dcf227af6117e: Status 404 returned error can't find the container with id 8d8f67dc938617ec74e2fd782e47b54c69dc64d36b78191bbb2dcf227af6117e
Jul 19 07:42:47.470255 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.468802    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb3a82c_68ca_4f01_9443_7467016acdb7.slice/crio-91b596b3f442ff59a3a2c8c55e2b344a665c5ecc588ef3d998331b501c0b40a4 WatchSource:0}: Error finding container 91b596b3f442ff59a3a2c8c55e2b344a665c5ecc588ef3d998331b501c0b40a4: Status 404 returned error can't find the container with id 91b596b3f442ff59a3a2c8c55e2b344a665c5ecc588ef3d998331b501c0b40a4
Jul 19 07:42:47.474120 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:42:47.473280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f2f7d_461c_40e1_9919_439c2dc8f02d.slice/crio-34902a1dc6df1b24fd8cd87762cccc6001abe8b6afa468cc4b6a6308ce07fc16 WatchSource:0}: Error finding container 34902a1dc6df1b24fd8cd87762cccc6001abe8b6afa468cc4b6a6308ce07fc16: Status 404 returned error can't find the container with id 34902a1dc6df1b24fd8cd87762cccc6001abe8b6afa468cc4b6a6308ce07fc16
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.953071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:47.953370    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.953658    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.953729    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:49.953708499 +0000 UTC m=+6.463306392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.954220    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:47.954360 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:47.954274    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:42:49.954257125 +0000 UTC m=+6.463855008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:48.054339 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.054302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:48.054531 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.054473    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:42:48.054531 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.054494    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:42:48.054531 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.054509    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:48.054697 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.054580    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:50.054561448 +0000 UTC m=+6.564159332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:48.368383 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.368318    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 07:37:46 +0000 UTC" deadline="2028-02-12 06:51:49.440933874 +0000 UTC"
Jul 19 07:42:48.368383 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.368380    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13751h9m1.072557823s"
Jul 19 07:42:48.455942 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.455909    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:48.456130 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.456050    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:48.456550 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.456508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:48.456677 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:48.456619    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:48.477427 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.477378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8z8rk" event={"ID":"f392427d-b56a-4351-b5a2-47d8dac7c6a5","Type":"ContainerStarted","Data":"7e659e5d51a76450da3da515eb99f9b98c816bef77db809e597000fef576e81f"}
Jul 19 07:42:48.489049 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.488993    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7wmg" event={"ID":"b47f2f7d-461c-40e1-9919-439c2dc8f02d","Type":"ContainerStarted","Data":"34902a1dc6df1b24fd8cd87762cccc6001abe8b6afa468cc4b6a6308ce07fc16"}
Jul 19 07:42:48.493093 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.493021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4njh" event={"ID":"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e","Type":"ContainerStarted","Data":"40a9413424563c019111318e7557387199ef5a82566587230a4ec0f7ab6044b6"}
Jul 19 07:42:48.501037 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.500974    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vx5wp" event={"ID":"1d332895-40aa-49e6-a1c2-33879f175472","Type":"ContainerStarted","Data":"8d8f67dc938617ec74e2fd782e47b54c69dc64d36b78191bbb2dcf227af6117e"}
Jul 19 07:42:48.513642 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.513541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc" event={"ID":"76520261-12a8-4ef9-b096-6a879578622b","Type":"ContainerStarted","Data":"b318d70176c31b7069eba0f91914740b940002181c9991afd48bd99163be6b32"}
Jul 19 07:42:48.528681 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.527995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal" event={"ID":"338760c60ab411de52b406a82ca30247","Type":"ContainerStarted","Data":"fad880d4a0ae42593cedd0c7c1709ce650c5d9e820d3a03b526f300071af1ebe"}
Jul 19 07:42:48.538570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.537942    2559 generic.go:356] "Generic (PLEG): container finished" podID="5459af162c0ff1eb3c23b44d0dcf199a" containerID="eccbf1c6b85bef3c1328a455406f72f08cf720bc1b198736f0b67e6243f1f4c5" exitCode=0
Jul 19 07:42:48.538570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.538024    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal" event={"ID":"5459af162c0ff1eb3c23b44d0dcf199a","Type":"ContainerDied","Data":"eccbf1c6b85bef3c1328a455406f72f08cf720bc1b198736f0b67e6243f1f4c5"}
Jul 19 07:42:48.543459 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.543156    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-91.ec2.internal" podStartSLOduration=2.543139128 podStartE2EDuration="2.543139128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:48.542979094 +0000 UTC m=+5.052576998" watchObservedRunningTime="2026-07-19 07:42:48.543139128 +0000 UTC m=+5.052737026"
Jul 19 07:42:48.558711 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.558642    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerStarted","Data":"fa8f4779cdd4209f8a5e3640d83f1254318f2ea1e4a777d11887665dce3132d9"}
Jul 19 07:42:48.572996 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.572965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-552d8" event={"ID":"0bb3a82c-68ca-4f01-9443-7467016acdb7","Type":"ContainerStarted","Data":"91b596b3f442ff59a3a2c8c55e2b344a665c5ecc588ef3d998331b501c0b40a4"}
Jul 19 07:42:48.578268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.578236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"c0d65790f06db7cb0dfd2a0ce7602218fdc6fe0ae34a8781723d3ed784359aa3"}
Jul 19 07:42:48.585815 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:48.584196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2dcs" event={"ID":"0a0eb070-3b10-43df-950b-f9ac992e9049","Type":"ContainerStarted","Data":"d425a9b08bd1835ed6431b53c33792775f526efd470873bf2ff01ea43b582e96"}
Jul 19 07:42:49.455175 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:49.455141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:49.455760 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:49.455287    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:49.608743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:49.608702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal" event={"ID":"5459af162c0ff1eb3c23b44d0dcf199a","Type":"ContainerStarted","Data":"b4713567b012e90e2cf941c376adabb68b0d63bb9ec405ec79b35cf424f71723"}
Jul 19 07:42:49.971135 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:49.971099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:49.971331 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:49.971193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:49.971331 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:49.971233    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:49.971331 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:49.971288    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:42:53.971273388 +0000 UTC m=+10.480871264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:49.971331 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:49.971305    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:49.971552 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:49.971359    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:53.971338696 +0000 UTC m=+10.480936590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:50.072260 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:50.072226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:50.072451 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.072406    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:42:50.072451 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.072427    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:42:50.072451 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.072439    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:50.072610 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.072496    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:42:54.072475429 +0000 UTC m=+10.582073309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:50.454489 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:50.454439    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:50.454662 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.454570    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:50.454986 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:50.454964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:50.455079 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:50.455063    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:51.455320 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:51.454839    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:51.455320 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:51.454975    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:52.455647 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:52.455604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:52.456106 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:52.455746    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:52.456180 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:52.456100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:52.456237 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:52.456221    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:53.455373 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:53.455335    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:53.455556 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:53.455477    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:54.006906 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:54.006871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:54.007380 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:54.006927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:54.007380 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.007028    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:54.007380 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.007091    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:43:02.007071832 +0000 UTC m=+18.516669733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:42:54.007380 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.007028    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:54.007380 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.007187    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:02.007164163 +0000 UTC m=+18.516762052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:42:54.107787 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:54.107751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:54.107961 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.107906    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:42:54.107961 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.107925    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:42:54.107961 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.107935    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:54.108135 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.107989    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:02.107973131 +0000 UTC m=+18.617571008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:42:54.458273 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:54.458218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:54.458273 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:54.458238    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:54.458506 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.458328    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:54.458506 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:54.458420    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:55.456004 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:55.455716    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:55.456004 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:55.455869    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:56.459433 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:56.458997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:56.459433 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:56.459115    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:56.459433 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:56.459258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:56.459433 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:56.459360    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:57.455434 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:57.455399    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:57.455600 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:57.455537    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:42:58.454880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:58.454792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:42:58.455311 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:58.454918    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:42:58.455311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:58.454953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:42:58.455311 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:58.455042    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:42:59.454716 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:42:59.454675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:42:59.454903 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:42:59.454824    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:00.455239 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:00.455203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:00.455726 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:00.455341    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:00.455726 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:00.455393    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:00.455726 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:00.455480    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:01.455233 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:01.455199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:01.455497 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:01.455349    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:02.069990 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:02.069965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:02.070089 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:02.070028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:02.070150 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.070125    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:43:02.070196 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.070186    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:43:18.070170977 +0000 UTC m=+34.579768867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:43:02.070243 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.070125    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:43:02.070281 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.070260    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:18.070247496 +0000 UTC m=+34.579845374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:43:02.171096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:02.171061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:02.171258 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.171242    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:43:02.171308 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.171271    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:43:02.171308 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.171282    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:43:02.171373 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.171363    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:18.171346942 +0000 UTC m=+34.680944838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:43:02.454789 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:02.454755    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:02.454967 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.454907    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:02.455022 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:02.454965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:02.455102 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:02.455080    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:03.454607 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:03.454575    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:03.455021 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:03.454731    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:04.455457 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:04.455426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:04.455945 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:04.455525    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:04.455945 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:04.455582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:04.455945 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:04.455657    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:05.454954 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:05.454919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:05.455162 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:05.455063    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:06.459855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.459052    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:06.459855 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:06.459342    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:06.459855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.459689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:06.459855 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:06.459801    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:06.654185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.653811    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="8cf3d8bfc63908b9e4589c733207c5f08b189c0f9d6c99eb9ec62ca6d8e41a71" exitCode=0
Jul 19 07:43:06.654185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.653904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"8cf3d8bfc63908b9e4589c733207c5f08b189c0f9d6c99eb9ec62ca6d8e41a71"}
Jul 19 07:43:06.656040 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.656013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"14f47bb916a2b8a921af8e4d84481e8726050c5aa4055ab99b08a2af8e670475"}
Jul 19 07:43:06.657676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.657652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2dcs" event={"ID":"0a0eb070-3b10-43df-950b-f9ac992e9049","Type":"ContainerStarted","Data":"c27ffc975c2aa10b16faa90e8da3d9391625a41787607c4a0e597225bc126c43"}
Jul 19 07:43:06.659173 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.659151    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8z8rk" event={"ID":"f392427d-b56a-4351-b5a2-47d8dac7c6a5","Type":"ContainerStarted","Data":"d6ebfd6f657ed8d0020d18eb5716513111a4648fa96fc6275c51876b35085d93"}
Jul 19 07:43:06.660633 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.660606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7wmg" event={"ID":"b47f2f7d-461c-40e1-9919-439c2dc8f02d","Type":"ContainerStarted","Data":"61228f4f52a3aebe87b6a66b772ce87c100364b5a470e5909067047733bfab61"}
Jul 19 07:43:06.662041 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.662014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b4njh" event={"ID":"760ff2a6-7b0c-4d1d-919a-b7e4029ef71e","Type":"ContainerStarted","Data":"425d45ba11f8cd9ec2fde1fa3e6beeaea79c69dacc4e6da4f85d705b84a7554f"}
Jul 19 07:43:06.663509 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.663485    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vx5wp" event={"ID":"1d332895-40aa-49e6-a1c2-33879f175472","Type":"ContainerStarted","Data":"a715279ee684d47224968c3efffaedb028473a0ae025b07553d0b07e67c0ea84"}
Jul 19 07:43:06.664925 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.664903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc" event={"ID":"76520261-12a8-4ef9-b096-6a879578622b","Type":"ContainerStarted","Data":"1b4afb09b35a9e573868b07a09cc85fc1f442a069f9b513d887654f94b296b28"}
Jul 19 07:43:06.679807 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.679769    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-91.ec2.internal" podStartSLOduration=20.679756654 podStartE2EDuration="20.679756654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:42:49.624645596 +0000 UTC m=+6.134243500" watchObservedRunningTime="2026-07-19 07:43:06.679756654 +0000 UTC m=+23.189354552"
Jul 19 07:43:06.699318 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.699270    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b4njh" podStartSLOduration=2.840892408 podStartE2EDuration="21.699251565s" totalImagesPullingTime="18.858359157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.699178074 +0000 UTC m=+23.208775972" watchObservedRunningTime="2026-07-19 07:43:06.699251565 +0000 UTC m=+23.208849464"
Jul 19 07:43:06.753014 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.752976    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2dcs" podStartSLOduration=3.863761561 podStartE2EDuration="22.752960745s" totalImagesPullingTime="18.889199184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.713401691 +0000 UTC m=+23.222999590" watchObservedRunningTime="2026-07-19 07:43:06.752960745 +0000 UTC m=+23.262558643"
Jul 19 07:43:06.753143 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.753065    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8z8rk" podStartSLOduration=2.855365872 podStartE2EDuration="21.75306158s" totalImagesPullingTime="18.897695708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.752804142 +0000 UTC m=+23.262402053" watchObservedRunningTime="2026-07-19 07:43:06.75306158 +0000 UTC m=+23.262659478"
Jul 19 07:43:06.780780 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.780740    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vx5wp" podStartSLOduration=3.9197513539999997 podStartE2EDuration="22.780726021s" totalImagesPullingTime="18.860974667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.780405218 +0000 UTC m=+23.290003116" watchObservedRunningTime="2026-07-19 07:43:06.780726021 +0000 UTC m=+23.290323910"
Jul 19 07:43:06.794944 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:06.794887    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m7wmg" podStartSLOduration=2.941665563 podStartE2EDuration="21.794866062s" totalImagesPullingTime="18.853200499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:06.794585029 +0000 UTC m=+23.304182930" watchObservedRunningTime="2026-07-19 07:43:06.794866062 +0000 UTC m=+23.304463962"
Jul 19 07:43:07.454950 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.454918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:07.455087 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:07.455064    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:07.669348 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.669315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-552d8" event={"ID":"0bb3a82c-68ca-4f01-9443-7467016acdb7","Type":"ContainerStarted","Data":"2bf075dd49984a367bcecc4286c43812f14714686417950a9408da3b309b0f73"}
Jul 19 07:43:07.673040 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.673011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"72d0cd9475533f088a4e06f7002d49710c1b251831298be38e7483b26992ce4a"}
Jul 19 07:43:07.673155 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.673092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"f97a16f550ca3ffebf9a8381a053cf11c963439517380ca41850cf93de481610"}
Jul 19 07:43:07.673155 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.673141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"302153ef0762083cde5c2d04057c1dba3a25a46f30152dbdb7737b77389b6e02"}
Jul 19 07:43:07.673266 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.673191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"2e4a05e1ad12b29991e3af6d8e875fdcc12ad84f6253e804d3daa10e41ef2dd2"}
Jul 19 07:43:07.673266 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.673235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"d5b392144c35e19735e34bdef672e95d443dc23adaba2e258ead0e6264454235"}
Jul 19 07:43:07.686949 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.686912    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-552d8" podStartSLOduration=3.820999299 podStartE2EDuration="22.686898441s" totalImagesPullingTime="18.865899142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:07.686315617 +0000 UTC m=+24.195913517" watchObservedRunningTime="2026-07-19 07:43:07.686898441 +0000 UTC m=+24.196496340"
Jul 19 07:43:07.727215 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:07.727166    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 19 07:43:08.393460 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.393341    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T07:43:07.72718621Z","UUID":"107194b0-4c90-4dd4-b143-efafc01a51a4","Handler":null,"Name":"","Endpoint":""}
Jul 19 07:43:08.395159 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.395132    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 19 07:43:08.395313 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.395174    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 19 07:43:08.458501 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.458472    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:08.458683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.458507    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:08.458683 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:08.458601    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:08.458823 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:08.458739    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:08.500541 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.500512    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:43:08.501281 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.501251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:43:08.676757 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.676674    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc" event={"ID":"76520261-12a8-4ef9-b096-6a879578622b","Type":"ContainerStarted","Data":"d1f637c504301d213bb2409cd0d0e1c90afa8c3acfe4ba810a89f01e86aed419"}
Jul 19 07:43:08.677226 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.676954    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:43:08.677442 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:08.677427    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vx5wp"
Jul 19 07:43:09.455434 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:09.455405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:09.455623 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:09.455555    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:09.681266 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:09.681234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc" event={"ID":"76520261-12a8-4ef9-b096-6a879578622b","Type":"ContainerStarted","Data":"b3b10f8ca623b7881b9304eb89e03f5392cc1bfa7aed71a4c8a429cb2f8f5d21"}
Jul 19 07:43:09.684961 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:09.684934    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"15d544bd381b8211fb6907c30fb99e095aed75d37c734d51e0320ab6120c0fc8"}
Jul 19 07:43:09.698200 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:09.698156    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-msptc" podStartSLOduration=4.729994256 podStartE2EDuration="25.698143254s" totalImagesPullingTime="20.968148998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:09.697881786 +0000 UTC m=+26.207479686" watchObservedRunningTime="2026-07-19 07:43:09.698143254 +0000 UTC m=+26.207741152"
Jul 19 07:43:10.455397 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:10.455189    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:10.455588 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:10.455301    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:10.455588 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:10.455520    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:10.455724 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:10.455586    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:11.455091 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.455068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:11.455524 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:11.455169    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:11.692044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.691832    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="3021d6a3b07695c016e89a4ec33e0fe681d58d5bc503e0f45fdeba0b8a6812a3" exitCode=0
Jul 19 07:43:11.692044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.691920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"3021d6a3b07695c016e89a4ec33e0fe681d58d5bc503e0f45fdeba0b8a6812a3"}
Jul 19 07:43:11.695735 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.695709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" event={"ID":"3e647a7e-bae7-4e90-97ee-a174eb876813","Type":"ContainerStarted","Data":"76c9cb1e77e3f091958a181eb6c4ed3247f662c8345fe58fd59bdbc6bff114b9"}
Jul 19 07:43:11.696052 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.696032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:11.710977 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.710959    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:11.734983 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:11.734944    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm" podStartSLOduration=7.776761829 podStartE2EDuration="26.734932423s" totalImagesPullingTime="18.958170594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:11.733539371 +0000 UTC m=+28.243137270" watchObservedRunningTime="2026-07-19 07:43:11.734932423 +0000 UTC m=+28.244530341"
Jul 19 07:43:12.455156 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:12.455126    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:12.455816 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:12.455245    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:12.455816 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:12.455275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:12.455816 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:12.455370    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:12.697450 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:12.697424    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 19 07:43:12.697841 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:12.697816    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:12.715219 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:12.713019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:13.454515 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:13.454477    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:13.454756 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:13.454583    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:13.700910 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:13.700880    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="ebba8e10a266011714c1c538d9eab96e18a81042dcbdb85ed39e8d47f6262e51" exitCode=0
Jul 19 07:43:13.701250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:13.700947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"ebba8e10a266011714c1c538d9eab96e18a81042dcbdb85ed39e8d47f6262e51"}
Jul 19 07:43:13.701250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:13.701009    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 19 07:43:14.457188 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:14.457160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:14.457358 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:14.457244    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:14.457358 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:14.457297    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:14.457438 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:14.457383    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:14.703468 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:14.703441    2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 19 07:43:14.761558 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:14.761525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:15.454520 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:15.454489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:15.454676 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:15.454603    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:15.707917 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:15.707843    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="6f239af9b1cabebf18acf4d0eee976579a4aa272751bd9316ac320d73f8475c0" exitCode=0
Jul 19 07:43:15.708261 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:15.707932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"6f239af9b1cabebf18acf4d0eee976579a4aa272751bd9316ac320d73f8475c0"}
Jul 19 07:43:16.455384 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:16.455351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:16.455384 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:16.455365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:16.455622 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:16.455470    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:16.455622 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:16.455588    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:17.454415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:17.454390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:17.454856 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:17.454522    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:18.095418 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:18.095385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:18.095592 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:18.095445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:18.095592 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.095551    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:43:18.095720 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.095630    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs podName:72fcf6a2-6fcc-4e80-b414-4d341f1e2719 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:50.095613348 +0000 UTC m=+66.605211239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs") pod "network-metrics-daemon-xnkds" (UID: "72fcf6a2-6fcc-4e80-b414-4d341f1e2719") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 19 07:43:18.095720 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.095557    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 19 07:43:18.095720 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.095694    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret podName:d90d5f08-c86a-42f2-a0d8-f9f316d0841f nodeName:}" failed. No retries permitted until 2026-07-19 07:43:50.095681683 +0000 UTC m=+66.605279565 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret") pod "global-pull-secret-syncer-p28rd" (UID: "d90d5f08-c86a-42f2-a0d8-f9f316d0841f") : object "kube-system"/"original-pull-secret" not registered
Jul 19 07:43:18.196661 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:18.196627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:18.196824 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.196784    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 19 07:43:18.196824 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.196804    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 19 07:43:18.196824 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.196813    2559 projected.go:196] Error preparing data for projected volume kube-api-access-5wvjm for pod openshift-network-diagnostics/network-check-target-qn6rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:43:18.196949 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.196873    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm podName:88684d25-db78-465f-b888-38f152f64a08 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:50.196853571 +0000 UTC m=+66.706451471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5wvjm" (UniqueName: "kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm") pod "network-check-target-qn6rj" (UID: "88684d25-db78-465f-b888-38f152f64a08") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 19 07:43:18.457984 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:18.457900    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:18.458410 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.458022    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:18.458410 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:18.458100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:18.458410 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:18.458212    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:19.454974 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:19.454942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:19.455156 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:19.455073    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:20.454632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:20.454561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:20.455052 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:20.454563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:20.455052 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:20.454658    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:20.455052 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:20.454751    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:20.721693 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:20.721610    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="0ce386bc776593799a72c72989f1816ca0d18ef83bf11ae8faedcbe535ced04b" exitCode=0
Jul 19 07:43:20.721839 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:20.721683    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"0ce386bc776593799a72c72989f1816ca0d18ef83bf11ae8faedcbe535ced04b"}
Jul 19 07:43:21.455299 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:21.455269    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:21.455653 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:21.455368    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:21.726828 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:21.726756    2559 generic.go:356] "Generic (PLEG): container finished" podID="5daba15c-5b35-477d-9154-ad2c0f2a108a" containerID="4409fc4fd01aeb49883e4903ea648c429a6dc18e614b045a8322471ef88daec0" exitCode=0
Jul 19 07:43:21.726943 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:21.726814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerDied","Data":"4409fc4fd01aeb49883e4903ea648c429a6dc18e614b045a8322471ef88daec0"}
Jul 19 07:43:22.455169 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:22.455137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:22.455336 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:22.455261    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:22.455336 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:22.455306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:22.455635 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:22.455389    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:22.732596 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:22.732522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" event={"ID":"5daba15c-5b35-477d-9154-ad2c0f2a108a","Type":"ContainerStarted","Data":"2ae810cae7c5b41da6c36095a62d94e68ef1f5b351c9017a85f80e76a0c0c620"}
Jul 19 07:43:22.756751 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:22.756713    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jrn9b" podStartSLOduration=7.865912392 podStartE2EDuration="37.756686175s" totalImagesPullingTime="29.890773783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:22.756365541 +0000 UTC m=+39.265963440" watchObservedRunningTime="2026-07-19 07:43:22.756686175 +0000 UTC m=+39.266284074"
Jul 19 07:43:23.455096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:23.455069    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:23.455277 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:23.455169    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:24.456253 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.456096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:24.456690 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:24.456305    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:24.456690 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.456166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:24.456690 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:24.456426    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:24.680962 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.680925    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p28rd"]
Jul 19 07:43:24.684245 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.684189    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xnkds"]
Jul 19 07:43:24.684380 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.684319    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:24.684481 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:24.684440    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:24.684913 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.684889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qn6rj"]
Jul 19 07:43:24.737629 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.737563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:24.737754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:24.737563    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:24.737754 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:24.737657    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:24.737754 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:24.737743    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:26.458402 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:26.458375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:26.459020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:26.458461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:26.459020 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:26.458471    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p28rd" podUID="d90d5f08-c86a-42f2-a0d8-f9f316d0841f"
Jul 19 07:43:26.459020 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:26.458546    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qn6rj" podUID="88684d25-db78-465f-b888-38f152f64a08"
Jul 19 07:43:26.459020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:26.458555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:26.459020 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:26.458665    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xnkds" podUID="72fcf6a2-6fcc-4e80-b414-4d341f1e2719"
Jul 19 07:43:28.292530 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.292458    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-91.ec2.internal" event="NodeReady"
Jul 19 07:43:28.293044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.292560    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 19 07:43:28.332460 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.332432    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v5fdl"]
Jul 19 07:43:28.354751 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.354724    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rrlzv"]
Jul 19 07:43:28.354896 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.354792    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.357317 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.357298    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 19 07:43:28.357464 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.357448    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 19 07:43:28.357541 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.357527    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vcblr\""
Jul 19 07:43:28.377228 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.377204    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-metrics-tls\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.377332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.377241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-config-volume\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.377332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.377261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-tmp-dir\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.377332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.377284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svtq\" (UniqueName: \"kubernetes.io/projected/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-kube-api-access-7svtq\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.378879 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.378862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5fdl"]
Jul 19 07:43:28.378924 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.378894    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrlzv"]
Jul 19 07:43:28.378924 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.378909    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ztphh"]
Jul 19 07:43:28.378985 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.378966    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.381624 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.381602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 19 07:43:28.381739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.381658    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qmw7p\""
Jul 19 07:43:28.381739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.381658    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 19 07:43:28.381739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.381662    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 19 07:43:28.396848 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.396826    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ztphh"]
Jul 19 07:43:28.396941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.396927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.399287 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.399269    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 19 07:43:28.399394 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.399358    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 19 07:43:28.399394 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.399377    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mhks2\""
Jul 19 07:43:28.399514 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.399360    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 19 07:43:28.399514 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.399473    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 19 07:43:28.454601 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.454576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:28.454725 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.454601    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:28.454725 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.454580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:28.457579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.457351    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 19 07:43:28.457579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.457375    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 19 07:43:28.457579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.457382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\""
Jul 19 07:43:28.457579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.457375    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 19 07:43:28.457579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.457458    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\""
Jul 19 07:43:28.464667 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.464648    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 19 07:43:28.478080 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.478174 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-cert\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.478174 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-metrics-tls\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.478267 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-config-volume\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.478267 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478205    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr7pd\" (UniqueName: \"kubernetes.io/projected/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-kube-api-access-gr7pd\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.478267 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-tmp-dir\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.478267 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svtq\" (UniqueName: \"kubernetes.io/projected/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-kube-api-access-7svtq\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.478415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478282    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-crio-socket\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.478415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-data-volume\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.478415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.478415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478362    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wclxk\" (UniqueName: \"kubernetes.io/projected/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-api-access-wclxk\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.478599 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478581    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-tmp-dir\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.478806 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.478785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-config-volume\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.481966 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.481946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-metrics-tls\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.485384 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.485361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svtq\" (UniqueName: \"kubernetes.io/projected/c686f16d-51d4-41e0-8096-e4c3ebf6d36a-kube-api-access-7svtq\") pod \"dns-default-v5fdl\" (UID: \"c686f16d-51d4-41e0-8096-e4c3ebf6d36a\") " pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.579622 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579588    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.579784 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-cert\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.579784 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr7pd\" (UniqueName: \"kubernetes.io/projected/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-kube-api-access-gr7pd\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.579784 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-crio-socket\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.579784 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-data-volume\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.579954 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.579954 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579828    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wclxk\" (UniqueName: \"kubernetes.io/projected/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-api-access-wclxk\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.580056 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.579956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-crio-socket\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.580186 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.580149    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-data-volume\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.580427 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.580409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.581934 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.581915    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.582139 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.582120    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-cert\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.588051 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.588028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wclxk\" (UniqueName: \"kubernetes.io/projected/f708f7ac-d9ce-4aca-9dd4-0d144f8eb126-kube-api-access-wclxk\") pod \"insights-runtime-extractor-ztphh\" (UID: \"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126\") " pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.588138 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.588048    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr7pd\" (UniqueName: \"kubernetes.io/projected/15106dd6-44d1-474a-9c3c-d22f30a7d1b6-kube-api-access-gr7pd\") pod \"ingress-canary-rrlzv\" (UID: \"15106dd6-44d1-474a-9c3c-d22f30a7d1b6\") " pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.664050 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.664016    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:28.687264 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.687232    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rrlzv"
Jul 19 07:43:28.704624 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.704606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ztphh"
Jul 19 07:43:28.861436 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.861406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v5fdl"]
Jul 19 07:43:28.865052 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:28.865026    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc686f16d_51d4_41e0_8096_e4c3ebf6d36a.slice/crio-ca629b7fddb5c41ad6bc63192f9327a2d5a5cd76f5a079aa0f7335372062e1a3 WatchSource:0}: Error finding container ca629b7fddb5c41ad6bc63192f9327a2d5a5cd76f5a079aa0f7335372062e1a3: Status 404 returned error can't find the container with id ca629b7fddb5c41ad6bc63192f9327a2d5a5cd76f5a079aa0f7335372062e1a3
Jul 19 07:43:28.865283 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.865251    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rrlzv"]
Jul 19 07:43:28.867807 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:28.867785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15106dd6_44d1_474a_9c3c_d22f30a7d1b6.slice/crio-d62df16e04cbcc5bddde9c96afc10e9cf062ca1373b007529fac796c3e1ea9fb WatchSource:0}: Error finding container d62df16e04cbcc5bddde9c96afc10e9cf062ca1373b007529fac796c3e1ea9fb: Status 404 returned error can't find the container with id d62df16e04cbcc5bddde9c96afc10e9cf062ca1373b007529fac796c3e1ea9fb
Jul 19 07:43:28.881811 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:28.881783    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ztphh"]
Jul 19 07:43:28.885725 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:28.885689    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf708f7ac_d9ce_4aca_9dd4_0d144f8eb126.slice/crio-5829fe86d99bafc50d522937fc37913af90d80c0d3709fd28ce95d80053c5156 WatchSource:0}: Error finding container 5829fe86d99bafc50d522937fc37913af90d80c0d3709fd28ce95d80053c5156: Status 404 returned error can't find the container with id 5829fe86d99bafc50d522937fc37913af90d80c0d3709fd28ce95d80053c5156
Jul 19 07:43:29.751642 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:29.751600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztphh" event={"ID":"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126","Type":"ContainerStarted","Data":"5829fe86d99bafc50d522937fc37913af90d80c0d3709fd28ce95d80053c5156"}
Jul 19 07:43:29.753406 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:29.753376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrlzv" event={"ID":"15106dd6-44d1-474a-9c3c-d22f30a7d1b6","Type":"ContainerStarted","Data":"d62df16e04cbcc5bddde9c96afc10e9cf062ca1373b007529fac796c3e1ea9fb"}
Jul 19 07:43:29.755259 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:29.755228    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5fdl" event={"ID":"c686f16d-51d4-41e0-8096-e4c3ebf6d36a","Type":"ContainerStarted","Data":"ca629b7fddb5c41ad6bc63192f9327a2d5a5cd76f5a079aa0f7335372062e1a3"}
Jul 19 07:43:35.771962 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.771781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztphh" event={"ID":"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126","Type":"ContainerStarted","Data":"eabbd08d1ea1c641c4dcc0fa01b539c63fa58d7c6b9f3bf6d874d1ac399b5884"}
Jul 19 07:43:35.773235 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.773210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rrlzv" event={"ID":"15106dd6-44d1-474a-9c3c-d22f30a7d1b6","Type":"ContainerStarted","Data":"aafe505b9a9bcb139875571b71caac6b62e67a51ef6d6a3c92b8a82bb3b2862d"}
Jul 19 07:43:35.774872 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.774847    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5fdl" event={"ID":"c686f16d-51d4-41e0-8096-e4c3ebf6d36a","Type":"ContainerStarted","Data":"e92b15421b9a8d7ed8218e9e87e6919afcc755af0e3106dfc1660cafd85ca536"}
Jul 19 07:43:35.774981 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.774881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v5fdl" event={"ID":"c686f16d-51d4-41e0-8096-e4c3ebf6d36a","Type":"ContainerStarted","Data":"aa45eb7949aac8c16e052ff8c82738307127ec9c4dec5fd1dcf636e36e20e755"}
Jul 19 07:43:35.778309 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.774928    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:35.791000 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.790958    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rrlzv" podStartSLOduration=1.421150929 podStartE2EDuration="7.790945974s" totalImagesPullingTime="6.369795045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:35.789841783 +0000 UTC m=+52.299439682" watchObservedRunningTime="2026-07-19 07:43:35.790945974 +0000 UTC m=+52.300543899"
Jul 19 07:43:35.804946 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:35.804755    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v5fdl" podStartSLOduration=1.432217201 podStartE2EDuration="7.804739767s" totalImagesPullingTime="6.372522566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:35.804628403 +0000 UTC m=+52.314226310" watchObservedRunningTime="2026-07-19 07:43:35.804739767 +0000 UTC m=+52.314337666"
Jul 19 07:43:37.411361 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.411326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"]
Jul 19 07:43:37.485636 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.485605    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-cxgdv"]
Jul 19 07:43:37.485792 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.485774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:37.488266 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.488241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sxhpq\""
Jul 19 07:43:37.495744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.495722    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 19 07:43:37.509170 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.509148    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"]
Jul 19 07:43:37.509279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.509179    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-cxgdv"]
Jul 19 07:43:37.509279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.509263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:37.512063 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.511803    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7kzv2\""
Jul 19 07:43:37.512063 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.511810    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 19 07:43:37.512063 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.511851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 19 07:43:37.554235 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.554202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09a35b42-9b4b-40e9-bc01-47854539ab1b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kvc67\" (UID: \"09a35b42-9b4b-40e9-bc01-47854539ab1b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:37.554388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.554313    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ddm\" (UniqueName: \"kubernetes.io/projected/88758f53-57e2-449e-8731-ee76a1f8efb9-kube-api-access-82ddm\") pod \"downloads-5bc67dcd46-cxgdv\" (UID: \"88758f53-57e2-449e-8731-ee76a1f8efb9\") " pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:37.655121 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.655076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ddm\" (UniqueName: \"kubernetes.io/projected/88758f53-57e2-449e-8731-ee76a1f8efb9-kube-api-access-82ddm\") pod \"downloads-5bc67dcd46-cxgdv\" (UID: \"88758f53-57e2-449e-8731-ee76a1f8efb9\") " pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:37.655347 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.655324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09a35b42-9b4b-40e9-bc01-47854539ab1b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kvc67\" (UID: \"09a35b42-9b4b-40e9-bc01-47854539ab1b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:37.659829 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.659802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/09a35b42-9b4b-40e9-bc01-47854539ab1b-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-kvc67\" (UID: \"09a35b42-9b4b-40e9-bc01-47854539ab1b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:37.663313 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.663230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ddm\" (UniqueName: \"kubernetes.io/projected/88758f53-57e2-449e-8731-ee76a1f8efb9-kube-api-access-82ddm\") pod \"downloads-5bc67dcd46-cxgdv\" (UID: \"88758f53-57e2-449e-8731-ee76a1f8efb9\") " pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:37.795999 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.795967    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:37.818627 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:37.818403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:39.445485 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:39.445457    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-cxgdv"]
Jul 19 07:43:39.450439 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:39.450412    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88758f53_57e2_449e_8731_ee76a1f8efb9.slice/crio-9c453c0eeca6116ebea1ad6597d0a8893b6fde783b8460152778187a0e8bc36e WatchSource:0}: Error finding container 9c453c0eeca6116ebea1ad6597d0a8893b6fde783b8460152778187a0e8bc36e: Status 404 returned error can't find the container with id 9c453c0eeca6116ebea1ad6597d0a8893b6fde783b8460152778187a0e8bc36e
Jul 19 07:43:39.452305 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:39.452278    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"]
Jul 19 07:43:39.458374 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:39.458340    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a35b42_9b4b_40e9_bc01_47854539ab1b.slice/crio-d37aad3cc9a775829c9a48228cd04a24d2d6f2c1ab3cb4a29f6b3eb20ff44f8a WatchSource:0}: Error finding container d37aad3cc9a775829c9a48228cd04a24d2d6f2c1ab3cb4a29f6b3eb20ff44f8a: Status 404 returned error can't find the container with id d37aad3cc9a775829c9a48228cd04a24d2d6f2c1ab3cb4a29f6b3eb20ff44f8a
Jul 19 07:43:39.787152 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:39.787121    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67" event={"ID":"09a35b42-9b4b-40e9-bc01-47854539ab1b","Type":"ContainerStarted","Data":"d37aad3cc9a775829c9a48228cd04a24d2d6f2c1ab3cb4a29f6b3eb20ff44f8a"}
Jul 19 07:43:39.788231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:39.788196    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-cxgdv" event={"ID":"88758f53-57e2-449e-8731-ee76a1f8efb9","Type":"ContainerStarted","Data":"9c453c0eeca6116ebea1ad6597d0a8893b6fde783b8460152778187a0e8bc36e"}
Jul 19 07:43:39.789806 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:39.789786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztphh" event={"ID":"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126","Type":"ContainerStarted","Data":"dcccbed5d38eaa3e995bfaddf2fc7aee2194acea08ed33522d48fd94635eab6b"}
Jul 19 07:43:41.799126 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:41.799086    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67" event={"ID":"09a35b42-9b4b-40e9-bc01-47854539ab1b","Type":"ContainerStarted","Data":"7e4782a2635ce82859735db8b2b1f1efd05ba2d00682b45426643b3846a3fd3d"}
Jul 19 07:43:41.799584 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:41.799321    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:41.805214 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:41.805188    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67"
Jul 19 07:43:41.813450 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:41.813409    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-kvc67" podStartSLOduration=3.192913313 podStartE2EDuration="4.813398052s" totalImagesPullingTime="1.620484739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:41.812166981 +0000 UTC m=+58.321764880" watchObservedRunningTime="2026-07-19 07:43:41.813398052 +0000 UTC m=+58.322995950"
Jul 19 07:43:42.467825 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.467792    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"]
Jul 19 07:43:42.553215 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.553181    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"]
Jul 19 07:43:42.553381 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.553318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.556284 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.556183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rwvkn\""
Jul 19 07:43:42.556284 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.556210    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 19 07:43:42.556447 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.556183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 19 07:43:42.557094 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.557072    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 19 07:43:42.557451 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.557330    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 19 07:43:42.557451 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.557361    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 19 07:43:42.697261 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.697221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.697426 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.697271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6gg\" (UniqueName: \"kubernetes.io/projected/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-kube-api-access-jz6gg\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.697426 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.697376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.697525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.697508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.798053 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.797972    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.798053 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.798029    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6gg\" (UniqueName: \"kubernetes.io/projected/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-kube-api-access-jz6gg\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.798268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.798098    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.798268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.798173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.799217 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.799057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.803000 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.802823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.803000 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.802914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.807729 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.806742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6gg\" (UniqueName: \"kubernetes.io/projected/c0171d8f-66e5-42d9-84ae-fb9a09c33bf9-kube-api-access-jz6gg\") pod \"prometheus-operator-5945b5cfd-q9lxs\" (UID: \"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:42.865470 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:42.865435    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"
Jul 19 07:43:43.893887 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:43.893862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs"]
Jul 19 07:43:43.898062 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:43.898035    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0171d8f_66e5_42d9_84ae_fb9a09c33bf9.slice/crio-4502b4c97a5c3fb784434dfad92132e391c8bd5c7edd459a76a239be5966d717 WatchSource:0}: Error finding container 4502b4c97a5c3fb784434dfad92132e391c8bd5c7edd459a76a239be5966d717: Status 404 returned error can't find the container with id 4502b4c97a5c3fb784434dfad92132e391c8bd5c7edd459a76a239be5966d717
Jul 19 07:43:44.812461 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:44.812420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs" event={"ID":"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9","Type":"ContainerStarted","Data":"4502b4c97a5c3fb784434dfad92132e391c8bd5c7edd459a76a239be5966d717"}
Jul 19 07:43:44.814787 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:44.814753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ztphh" event={"ID":"f708f7ac-d9ce-4aca-9dd4-0d144f8eb126","Type":"ContainerStarted","Data":"c599e53512949ec4893684a0ad76381e1d975b7b1b0cf60b298c150b49896b2a"}
Jul 19 07:43:44.839125 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:44.839063    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ztphh" podStartSLOduration=2.142808858 podStartE2EDuration="16.839046107s" totalImagesPullingTime="14.696237249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:44.838212641 +0000 UTC m=+61.347810540" watchObservedRunningTime="2026-07-19 07:43:44.839046107 +0000 UTC m=+61.348644207"
Jul 19 07:43:45.720006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:45.719976    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qbtsm"
Jul 19 07:43:45.780744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:45.780715    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v5fdl"
Jul 19 07:43:45.821301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:45.821273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs" event={"ID":"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9","Type":"ContainerStarted","Data":"4a81f0afa6ad434b2d04c60015130d66cc998602b6a361d83d0d93bf5ac6df83"}
Jul 19 07:43:45.821478 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:45.821327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs" event={"ID":"c0171d8f-66e5-42d9-84ae-fb9a09c33bf9","Type":"ContainerStarted","Data":"5c1ba57d34f70eba360709b56eb829af409372f0db67b112115ae7aa7190c7ba"}
Jul 19 07:43:45.839986 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:45.839931    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-q9lxs" podStartSLOduration=2.5813816320000003 podStartE2EDuration="3.839917228s" totalImagesPullingTime="1.258535596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:45.838525099 +0000 UTC m=+62.348122997" watchObservedRunningTime="2026-07-19 07:43:45.839917228 +0000 UTC m=+62.349515126"
Jul 19 07:43:47.784398 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.784365    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"]
Jul 19 07:43:47.786653 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.786632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:47.788954 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.788932    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 19 07:43:47.789085 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.788994    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 19 07:43:47.789266 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.789252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sf7qq\""
Jul 19 07:43:47.792172 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.792137    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"]
Jul 19 07:43:47.794247 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.794230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.799206 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.799131    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 19 07:43:47.799206 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.799163    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-92nt6\""
Jul 19 07:43:47.800637 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.800617    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 19 07:43:47.800758 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.800623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 19 07:43:47.805093 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.805074    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"]
Jul 19 07:43:47.811279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.811256    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"]
Jul 19 07:43:47.830023 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.829981    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pfsv5"]
Jul 19 07:43:47.834370 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.834350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.837712 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.837676    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 19 07:43:47.837795 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.837738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 19 07:43:47.838085 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.838059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5pdwm\""
Jul 19 07:43:47.838178 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.838156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 19 07:43:47.948289 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948248    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9zkd\" (UniqueName: \"kubernetes.io/projected/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-api-access-x9zkd\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948289 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-textfile\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-sys\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-root\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948507    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-metrics-client-ca\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvxh\" (UniqueName: \"kubernetes.io/projected/86522d1a-24ec-499e-a1ed-6a36b9827c14-kube-api-access-rqvxh\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948620    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86522d1a-24ec-499e-a1ed-6a36b9827c14-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948651    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-wtmp\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7vfd\" (UniqueName: \"kubernetes.io/projected/f348102a-107a-4ab9-b00e-5b69957c0a41-kube-api-access-s7vfd\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735e1b8b-d9a4-40b1-ad84-7e668521baf6-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:47.948993 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:47.948806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7vfd\" (UniqueName: \"kubernetes.io/projected/f348102a-107a-4ab9-b00e-5b69957c0a41-kube-api-access-s7vfd\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735e1b8b-d9a4-40b1-ad84-7e668521baf6-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050281 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.050281 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050379 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9zkd\" (UniqueName: \"kubernetes.io/projected/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-api-access-x9zkd\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050379 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-textfile\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/735e1b8b-d9a4-40b1-ad84-7e668521baf6-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050476 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050473    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-sys\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-root\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050625    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-metrics-client-ca\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvxh\" (UniqueName: \"kubernetes.io/projected/86522d1a-24ec-499e-a1ed-6a36b9827c14-kube-api-access-rqvxh\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86522d1a-24ec-499e-a1ed-6a36b9827c14-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.050739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-textfile\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-wtmp\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:48.050828    2559 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-root\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:48.050884    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls podName:735e1b8b-d9a4-40b1-ad84-7e668521baf6 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:48.550866117 +0000 UTC m=+65.060464017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-nzwdq" (UID: "735e1b8b-d9a4-40b1-ad84-7e668521baf6") : secret "kube-state-metrics-tls" not found
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-sys\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050924    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-wtmp\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:48.050938    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.050954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.051097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.051185 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:48.050969    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls podName:f348102a-107a-4ab9-b00e-5b69957c0a41 nodeName:}" failed. No retries permitted until 2026-07-19 07:43:48.550958943 +0000 UTC m=+65.060556827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls") pod "node-exporter-pfsv5" (UID: "f348102a-107a-4ab9-b00e-5b69957c0a41") : secret "node-exporter-tls" not found
Jul 19 07:43:48.051761 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.051457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f348102a-107a-4ab9-b00e-5b69957c0a41-metrics-client-ca\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.051761 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.051641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86522d1a-24ec-499e-a1ed-6a36b9827c14-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.051956 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.051873    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.053577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.053550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.053763 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.053739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.053853 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.053762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.053853 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.053780    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/86522d1a-24ec-499e-a1ed-6a36b9827c14-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.063325 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.063299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7vfd\" (UniqueName: \"kubernetes.io/projected/f348102a-107a-4ab9-b00e-5b69957c0a41-kube-api-access-s7vfd\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.063415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.063383    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9zkd\" (UniqueName: \"kubernetes.io/projected/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-api-access-x9zkd\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.063505 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.063481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvxh\" (UniqueName: \"kubernetes.io/projected/86522d1a-24ec-499e-a1ed-6a36b9827c14-kube-api-access-rqvxh\") pod \"openshift-state-metrics-79f6958cdf-nlh7n\" (UID: \"86522d1a-24ec-499e-a1ed-6a36b9827c14\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.100263 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.100237    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"
Jul 19 07:43:48.238685 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.238617    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n"]
Jul 19 07:43:48.241420 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:48.241394    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86522d1a_24ec_499e_a1ed_6a36b9827c14.slice/crio-4a272879858669bb4f28f54747f5f0b5868811c650e7deec5201a9baa936100a WatchSource:0}: Error finding container 4a272879858669bb4f28f54747f5f0b5868811c650e7deec5201a9baa936100a: Status 404 returned error can't find the container with id 4a272879858669bb4f28f54747f5f0b5868811c650e7deec5201a9baa936100a
Jul 19 07:43:48.318650 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.318625    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:43:48.321145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.321116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.324006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.323583    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 19 07:43:48.324006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.323607    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 19 07:43:48.324006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.323641    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bb9gk\""
Jul 19 07:43:48.324006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.323880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 19 07:43:48.324577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.324549    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 19 07:43:48.324577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.324557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 19 07:43:48.332180 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.332136    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:43:48.455216 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.455216 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.455570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv47\" (UniqueName: \"kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.455570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.455570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.455570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.455414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.556461 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.556461 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.556775 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.556775 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.556991 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556940    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv47\" (UniqueName: \"kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.557076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.556986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.557076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.557019    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.557076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.557045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.557949 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.557255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.557949 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.557391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.557949 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.557895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.559404 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.559382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.559817 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.559794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/735e1b8b-d9a4-40b1-ad84-7e668521baf6-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-nzwdq\" (UID: \"735e1b8b-d9a4-40b1-ad84-7e668521baf6\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.559901 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.559839    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.560122 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.560103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f348102a-107a-4ab9-b00e-5b69957c0a41-node-exporter-tls\") pod \"node-exporter-pfsv5\" (UID: \"f348102a-107a-4ab9-b00e-5b69957c0a41\") " pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.566688 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.566668    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv47\" (UniqueName: \"kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47\") pod \"console-855f84d8c6-ns9g4\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") " pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.642999 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.642883    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:43:48.707345 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.707094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"
Jul 19 07:43:48.746396 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.746363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pfsv5"
Jul 19 07:43:48.802564 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.802239    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:43:48.806853 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:48.806793    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ac2fcb_4549_40f7_a9d9_fc4857f7213b.slice/crio-7d19ee6e6dd66db06bea553107ee99c5cc3b288f4e83c81c14004dab45cdee34 WatchSource:0}: Error finding container 7d19ee6e6dd66db06bea553107ee99c5cc3b288f4e83c81c14004dab45cdee34: Status 404 returned error can't find the container with id 7d19ee6e6dd66db06bea553107ee99c5cc3b288f4e83c81c14004dab45cdee34
Jul 19 07:43:48.840295 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.840258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfsv5" event={"ID":"f348102a-107a-4ab9-b00e-5b69957c0a41","Type":"ContainerStarted","Data":"cdb679c16f73a1a3f95719c4a8327e47b8c305e5a8b1c0f7bb79412f41534311"}
Jul 19 07:43:48.842405 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.842376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n" event={"ID":"86522d1a-24ec-499e-a1ed-6a36b9827c14","Type":"ContainerStarted","Data":"3caae798e02b85a2e2966f838a37de4d8b0025e28b1506ec372b9d19284f0528"}
Jul 19 07:43:48.842545 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.842442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n" event={"ID":"86522d1a-24ec-499e-a1ed-6a36b9827c14","Type":"ContainerStarted","Data":"357a0c99c0d9f7b051aeac037729441859cc9281363e1bed27886431c26c5d06"}
Jul 19 07:43:48.842545 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.842471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n" event={"ID":"86522d1a-24ec-499e-a1ed-6a36b9827c14","Type":"ContainerStarted","Data":"4a272879858669bb4f28f54747f5f0b5868811c650e7deec5201a9baa936100a"}
Jul 19 07:43:48.843654 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.843627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855f84d8c6-ns9g4" event={"ID":"66ac2fcb-4549-40f7-a9d9-fc4857f7213b","Type":"ContainerStarted","Data":"7d19ee6e6dd66db06bea553107ee99c5cc3b288f4e83c81c14004dab45cdee34"}
Jul 19 07:43:48.860878 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:48.860854    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-nzwdq"]
Jul 19 07:43:48.864161 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:48.864134    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735e1b8b_d9a4_40b1_ad84_7e668521baf6.slice/crio-36e505661fc4b1fa414c4f5de4a627bf36f4fb43414d95447a0615a9a740ef18 WatchSource:0}: Error finding container 36e505661fc4b1fa414c4f5de4a627bf36f4fb43414d95447a0615a9a740ef18: Status 404 returned error can't find the container with id 36e505661fc4b1fa414c4f5de4a627bf36f4fb43414d95447a0615a9a740ef18
Jul 19 07:43:49.849195 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:49.849028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq" event={"ID":"735e1b8b-d9a4-40b1-ad84-7e668521baf6","Type":"ContainerStarted","Data":"36e505661fc4b1fa414c4f5de4a627bf36f4fb43414d95447a0615a9a740ef18"}
Jul 19 07:43:50.171587 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.171444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:50.171587 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.171519    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:50.173962 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.173936    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 19 07:43:50.174098 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.174016    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 19 07:43:50.185752 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.185679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d90d5f08-c86a-42f2-a0d8-f9f316d0841f-original-pull-secret\") pod \"global-pull-secret-syncer-p28rd\" (UID: \"d90d5f08-c86a-42f2-a0d8-f9f316d0841f\") " pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:50.186810 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.186779    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72fcf6a2-6fcc-4e80-b414-4d341f1e2719-metrics-certs\") pod \"network-metrics-daemon-xnkds\" (UID: \"72fcf6a2-6fcc-4e80-b414-4d341f1e2719\") " pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:50.272898 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.272861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:50.275278 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.275228    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 19 07:43:50.285711 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.285664    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 19 07:43:50.297676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.297644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wvjm\" (UniqueName: \"kubernetes.io/projected/88684d25-db78-465f-b888-38f152f64a08-kube-api-access-5wvjm\") pod \"network-check-target-qn6rj\" (UID: \"88684d25-db78-465f-b888-38f152f64a08\") " pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:50.365669 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.365634    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ptfjq\""
Jul 19 07:43:50.370739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.370693    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrpxw\""
Jul 19 07:43:50.373715 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.373673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:43:50.373832 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.373688    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p28rd"
Jul 19 07:43:50.379480 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.379461    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xnkds"
Jul 19 07:43:50.856329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.856278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n" event={"ID":"86522d1a-24ec-499e-a1ed-6a36b9827c14","Type":"ContainerStarted","Data":"15852c2d385d7b1d1104685af8bf46db35417e3bfc06054ceb95dadea8c84ac2"}
Jul 19 07:43:50.874035 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:50.873984    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-nlh7n" podStartSLOduration=2.639741351 podStartE2EDuration="3.873969052s" totalImagesPullingTime="1.234227701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:50.872177049 +0000 UTC m=+67.381774949" watchObservedRunningTime="2026-07-19 07:43:50.873969052 +0000 UTC m=+67.383566945"
Jul 19 07:43:52.584914 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.584881    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"]
Jul 19 07:43:52.587085 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.587065    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:52.589553 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.589524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 19 07:43:52.589650 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.589539    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xdfjm\""
Jul 19 07:43:52.595133 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.595109    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"]
Jul 19 07:43:52.694001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.693970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5m7s\" (UniqueName: \"kubernetes.io/projected/cfc6dd51-1b56-428b-b17b-1e757727ec3a-kube-api-access-r5m7s\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:52.694192 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.694036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:52.794896 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.794861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:52.795078 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.794917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5m7s\" (UniqueName: \"kubernetes.io/projected/cfc6dd51-1b56-428b-b17b-1e757727ec3a-kube-api-access-r5m7s\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:52.795078 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:52.795030    2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 19 07:43:52.795195 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:43:52.795099    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert podName:cfc6dd51-1b56-428b-b17b-1e757727ec3a nodeName:}" failed. No retries permitted until 2026-07-19 07:43:53.29507945 +0000 UTC m=+69.804677327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-84bhj" (UID: "cfc6dd51-1b56-428b-b17b-1e757727ec3a") : secret "monitoring-plugin-cert" not found
Jul 19 07:43:52.803666 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:52.803633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5m7s\" (UniqueName: \"kubernetes.io/projected/cfc6dd51-1b56-428b-b17b-1e757727ec3a-kube-api-access-r5m7s\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:53.298713 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:53.298658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:53.301287 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:53.301261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cfc6dd51-1b56-428b-b17b-1e757727ec3a-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-84bhj\" (UID: \"cfc6dd51-1b56-428b-b17b-1e757727ec3a\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:53.499602 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:53.499564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:43:55.908349 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:55.908314    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:43:55.939083 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:55.939047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:43:55.939231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:55.939144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:55.947600 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:55.947568    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 19 07:43:56.119632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119820 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119820 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvcp8\" (UniqueName: \"kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119820 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119984 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119984 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119882    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.119984 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.119914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.220946 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.220857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.220946 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.220919    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvcp8\" (UniqueName: \"kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221177 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.220951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221177 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.221006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221177 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.221043    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221177 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.221072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221177 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.221111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.221751 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.221720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.222132 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.222104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.222248 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.222107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.222748 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.222725    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.223742 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.223718    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.223874 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.223854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.229063 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.229039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvcp8\" (UniqueName: \"kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8\") pod \"console-847b7b8f78-th72n\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") " pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:56.252981 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:56.252958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:43:59.815571 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.815517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qn6rj"]
Jul 19 07:43:59.877450 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.877224    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"]
Jul 19 07:43:59.881117 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:59.881076    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc6dd51_1b56_428b_b17b_1e757727ec3a.slice/crio-a3ec6deca92dbf881a9e0b2c92e71198dde3111e820894adc7934a80b2f742ee WatchSource:0}: Error finding container a3ec6deca92dbf881a9e0b2c92e71198dde3111e820894adc7934a80b2f742ee: Status 404 returned error can't find the container with id a3ec6deca92dbf881a9e0b2c92e71198dde3111e820894adc7934a80b2f742ee
Jul 19 07:43:59.905507 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.905427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfsv5" event={"ID":"f348102a-107a-4ab9-b00e-5b69957c0a41","Type":"ContainerStarted","Data":"95f53e8369280e175a0c459f1dc3071b33ff12f1c1a4458e7765d55019107e1d"}
Jul 19 07:43:59.908076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.907844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-cxgdv" event={"ID":"88758f53-57e2-449e-8731-ee76a1f8efb9","Type":"ContainerStarted","Data":"d17fd9710265c368c30d9590ea45bf40329c5f26053bc19daf3f1ec41de3b715"}
Jul 19 07:43:59.908732 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.908570    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:43:59.912149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.910440    2559 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-cxgdv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body=
Jul 19 07:43:59.912149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.911313    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-cxgdv" podUID="88758f53-57e2-449e-8731-ee76a1f8efb9" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused"
Jul 19 07:43:59.916937 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.916882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xnkds"]
Jul 19 07:43:59.918683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.918594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq" event={"ID":"735e1b8b-d9a4-40b1-ad84-7e668521baf6","Type":"ContainerStarted","Data":"68da40c3d19242c0644a8afa32d2795ae5bc357fbe1d27bff806dd3e41951651"}
Jul 19 07:43:59.918683 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.918645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq" event={"ID":"735e1b8b-d9a4-40b1-ad84-7e668521baf6","Type":"ContainerStarted","Data":"44062b093e89964169a2e7ab07320d1e3dcd220f00b6f7aebd1247a0eb4266bc"}
Jul 19 07:43:59.924658 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:43:59.922802    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fcf6a2_6fcc_4e80_b414_4d341f1e2719.slice/crio-97be82a21da09d830eac1dc09710f7afddf47ce8581d3487460402bd37abda33 WatchSource:0}: Error finding container 97be82a21da09d830eac1dc09710f7afddf47ce8581d3487460402bd37abda33: Status 404 returned error can't find the container with id 97be82a21da09d830eac1dc09710f7afddf47ce8581d3487460402bd37abda33
Jul 19 07:43:59.924658 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.923273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855f84d8c6-ns9g4" event={"ID":"66ac2fcb-4549-40f7-a9d9-fc4857f7213b","Type":"ContainerStarted","Data":"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"}
Jul 19 07:43:59.933004 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.932958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qn6rj" event={"ID":"88684d25-db78-465f-b888-38f152f64a08","Type":"ContainerStarted","Data":"766cccf674e58d8d2c32dd6feba9f89d60a33e3b0d13f52198f06d63302883c1"}
Jul 19 07:43:59.937551 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.937445    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj" event={"ID":"cfc6dd51-1b56-428b-b17b-1e757727ec3a","Type":"ContainerStarted","Data":"a3ec6deca92dbf881a9e0b2c92e71198dde3111e820894adc7934a80b2f742ee"}
Jul 19 07:43:59.950006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:43:59.949813    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-855f84d8c6-ns9g4" podStartSLOduration=1.1437503310000001 podStartE2EDuration="11.949793031s" totalImagesPullingTime="10.8060427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:59.949051377 +0000 UTC m=+76.458649281" watchObservedRunningTime="2026-07-19 07:43:59.949793031 +0000 UTC m=+76.459390933"
Jul 19 07:44:00.046223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.045532    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-cxgdv" podStartSLOduration=2.890426299 podStartE2EDuration="23.045505449s" totalImagesPullingTime="20.15507915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:43:59.963764521 +0000 UTC m=+76.473362421" watchObservedRunningTime="2026-07-19 07:44:00.045505449 +0000 UTC m=+76.555103348"
Jul 19 07:44:00.047759 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.046847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p28rd"]
Jul 19 07:44:00.049421 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.048941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:44:00.947722 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.947613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfsv5" event={"ID":"f348102a-107a-4ab9-b00e-5b69957c0a41","Type":"ContainerDied","Data":"95f53e8369280e175a0c459f1dc3071b33ff12f1c1a4458e7765d55019107e1d"}
Jul 19 07:44:00.948253 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.947726    2559 generic.go:356] "Generic (PLEG): container finished" podID="f348102a-107a-4ab9-b00e-5b69957c0a41" containerID="95f53e8369280e175a0c459f1dc3071b33ff12f1c1a4458e7765d55019107e1d" exitCode=0
Jul 19 07:44:00.953672 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.953384    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq" event={"ID":"735e1b8b-d9a4-40b1-ad84-7e668521baf6","Type":"ContainerStarted","Data":"142b9a4f29e4e3b90d447d3e19f2a8dc5b01a5d1d0eee0925ba01fd7c9f7cf98"}
Jul 19 07:44:00.955681 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.955621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnkds" event={"ID":"72fcf6a2-6fcc-4e80-b414-4d341f1e2719","Type":"ContainerStarted","Data":"97be82a21da09d830eac1dc09710f7afddf47ce8581d3487460402bd37abda33"}
Jul 19 07:44:00.958584 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.958526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p28rd" event={"ID":"d90d5f08-c86a-42f2-a0d8-f9f316d0841f","Type":"ContainerStarted","Data":"def3a756892d0fb01b17462ca4a962685d04c4a5f30471731de728c8a9022dc4"}
Jul 19 07:44:00.961885 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.960752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b7b8f78-th72n" event={"ID":"42bd4eab-8bc3-426b-8884-950cb099b0c9","Type":"ContainerStarted","Data":"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"}
Jul 19 07:44:00.961885 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.960789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b7b8f78-th72n" event={"ID":"42bd4eab-8bc3-426b-8884-950cb099b0c9","Type":"ContainerStarted","Data":"ed6b8e033e723127d96d6770ca01ae54ec0ab6829c84b84ee158f1dcea2f5157"}
Jul 19 07:44:00.973833 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.973713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-cxgdv"
Jul 19 07:44:00.985915 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:00.985809    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-nzwdq" podStartSLOduration=3.244452718 podStartE2EDuration="13.98578907s" totalImagesPullingTime="10.741336352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:00.984576321 +0000 UTC m=+77.494174221" watchObservedRunningTime="2026-07-19 07:44:00.98578907 +0000 UTC m=+77.495386970"
Jul 19 07:44:01.003421 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:01.002806    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-847b7b8f78-th72n" podStartSLOduration=6.002785968 podStartE2EDuration="6.002785968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:01.001238457 +0000 UTC m=+77.510836358" watchObservedRunningTime="2026-07-19 07:44:01.002785968 +0000 UTC m=+77.512383862"
Jul 19 07:44:03.990043 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.989467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj" event={"ID":"cfc6dd51-1b56-428b-b17b-1e757727ec3a","Type":"ContainerStarted","Data":"34d6ed5c1da1aa58cafb183f48978abdfc17607237c973292f31cd5d4590d24e"}
Jul 19 07:44:03.990043 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.989730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:44:03.994062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.993960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfsv5" event={"ID":"f348102a-107a-4ab9-b00e-5b69957c0a41","Type":"ContainerStarted","Data":"5fdf8c8c9f34c504a21d4074fb51da40fd586de0c99d0bd3e2d20f1023106f83"}
Jul 19 07:44:03.994062 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.994011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pfsv5" event={"ID":"f348102a-107a-4ab9-b00e-5b69957c0a41","Type":"ContainerStarted","Data":"40a8f142dfdcbb47024ac9016ab1741d01983fe2ace14d642c6967a7a5ae8bd0"}
Jul 19 07:44:03.996403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.996326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnkds" event={"ID":"72fcf6a2-6fcc-4e80-b414-4d341f1e2719","Type":"ContainerStarted","Data":"4e4d49779561c0fc10eef61154fe8e225b88feafae8eba4550b88919cdf52e3f"}
Jul 19 07:44:03.996403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:03.996371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xnkds" event={"ID":"72fcf6a2-6fcc-4e80-b414-4d341f1e2719","Type":"ContainerStarted","Data":"44e8e16192f2cac52ea29dd71c0192b08a60b440763ce4b1736407f4a5bac224"}
Jul 19 07:44:04.005890 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:04.005768    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj" podStartSLOduration=8.839672604 podStartE2EDuration="12.005741616s" totalImagesPullingTime="3.166069012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:43:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:04.004958038 +0000 UTC m=+80.514555949" watchObservedRunningTime="2026-07-19 07:44:04.005741616 +0000 UTC m=+80.515339517"
Jul 19 07:44:04.006480 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:04.006461    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-84bhj"
Jul 19 07:44:04.022004 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:04.021179    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xnkds" podStartSLOduration=75.90593795 podStartE2EDuration="1m19.021165738s" totalImagesPullingTime="3.115227788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:04.020881367 +0000 UTC m=+80.530479267" watchObservedRunningTime="2026-07-19 07:44:04.021165738 +0000 UTC m=+80.530763638"
Jul 19 07:44:04.040525 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:04.040471    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pfsv5" podStartSLOduration=6.066498411 podStartE2EDuration="17.040454431s" totalImagesPullingTime="10.833480074s" totalInitContainerRuntime="140.475946ms" isStatefulPod=true podCreationTimestamp="2026-07-19 07:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:04.039917474 +0000 UTC m=+80.549515399" watchObservedRunningTime="2026-07-19 07:44:04.040454431 +0000 UTC m=+80.550052331"
Jul 19 07:44:06.253520 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:06.253367    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:44:06.253520 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:06.253478    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:44:06.259619 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:06.259593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:44:07.016098 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:07.016069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:44:07.059819 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:07.059788    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:44:08.018067 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.018029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qn6rj" event={"ID":"88684d25-db78-465f-b888-38f152f64a08","Type":"ContainerStarted","Data":"88bafb0c7895a0d213df25d043e39a06cb71dfc583c15163534ee69ec16eea67"}
Jul 19 07:44:08.018603 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.018114    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:44:08.020636 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.020606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p28rd" event={"ID":"d90d5f08-c86a-42f2-a0d8-f9f316d0841f","Type":"ContainerStarted","Data":"664875acd4886115a847f1deb9e1102df5c14ad0607ffa75edef030794fea09d"}
Jul 19 07:44:08.034498 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.034306    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qn6rj" podStartSLOduration=76.646580141 podStartE2EDuration="1m24.034287571s" totalImagesPullingTime="7.38770743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:08.031461063 +0000 UTC m=+84.541058966" watchObservedRunningTime="2026-07-19 07:44:08.034287571 +0000 UTC m=+84.543885471"
Jul 19 07:44:08.045246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.045206    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p28rd" podStartSLOduration=75.874182017 podStartE2EDuration="1m23.045193396s" totalImagesPullingTime="7.171011379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 07:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:44:08.044014759 +0000 UTC m=+84.553612657" watchObservedRunningTime="2026-07-19 07:44:08.045193396 +0000 UTC m=+84.554791294"
Jul 19 07:44:08.643310 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:08.643278    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:44:32.084542 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.084482    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-855f84d8c6-ns9g4" podUID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" containerName="console" containerID="cri-o://5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f" gracePeriod=15
Jul 19 07:44:32.351664 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.351600    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:44:32.407919 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.407890    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxv47\" (UniqueName: \"kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.407927    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.407963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408016    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408101 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408051    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408305 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408104    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config\") pod \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\" (UID: \"66ac2fcb-4549-40f7-a9d9-fc4857f7213b\") "
Jul 19 07:44:32.408459 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408424    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca" (OuterVolumeSpecName: "service-ca") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:44:32.408568 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408457    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config" (OuterVolumeSpecName: "console-config") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:44:32.408568 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.408436    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:44:32.410445 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.410411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:44:32.410574 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.410553    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:44:32.410629 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.410606    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47" (OuterVolumeSpecName: "kube-api-access-jxv47") pod "66ac2fcb-4549-40f7-a9d9-fc4857f7213b" (UID: "66ac2fcb-4549-40f7-a9d9-fc4857f7213b"). InnerVolumeSpecName "kube-api-access-jxv47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:44:32.508713 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508683    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:32.508826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508725    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-oauth-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:32.508826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508735    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxv47\" (UniqueName: \"kubernetes.io/projected/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-kube-api-access-jxv47\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:32.508826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508744    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-console-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:32.508826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508753    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:32.508826 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:32.508761    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66ac2fcb-4549-40f7-a9d9-fc4857f7213b-oauth-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:44:33.109248 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.109210    2559 generic.go:356] "Generic (PLEG): container finished" podID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" containerID="5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f" exitCode=0
Jul 19 07:44:33.109637 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.109264    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-855f84d8c6-ns9g4"
Jul 19 07:44:33.109637 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.109306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855f84d8c6-ns9g4" event={"ID":"66ac2fcb-4549-40f7-a9d9-fc4857f7213b","Type":"ContainerDied","Data":"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"}
Jul 19 07:44:33.109637 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.109361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-855f84d8c6-ns9g4" event={"ID":"66ac2fcb-4549-40f7-a9d9-fc4857f7213b","Type":"ContainerDied","Data":"7d19ee6e6dd66db06bea553107ee99c5cc3b288f4e83c81c14004dab45cdee34"}
Jul 19 07:44:33.109637 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.109382    2559 scope.go:117] "RemoveContainer" containerID="5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"
Jul 19 07:44:33.119421 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.119404    2559 scope.go:117] "RemoveContainer" containerID="5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"
Jul 19 07:44:33.119688 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:44:33.119670    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f\": container with ID starting with 5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f not found: ID does not exist" containerID="5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"
Jul 19 07:44:33.119762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.119711    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f"} err="failed to get container status \"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f\": rpc error: code = NotFound desc = could not find container \"5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f\": container with ID starting with 5776bf056a33364002c22e1424ab18f84bcd3b441d1ddc886f49e59ca220550f not found: ID does not exist"
Jul 19 07:44:33.126448 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.126423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:44:33.132800 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:33.132779    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-855f84d8c6-ns9g4"]
Jul 19 07:44:34.459307 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:34.459232    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" path="/var/lib/kubelet/pods/66ac2fcb-4549-40f7-a9d9-fc4857f7213b/volumes"
Jul 19 07:44:39.026868 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:44:39.026835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qn6rj"
Jul 19 07:45:12.035087 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.035042    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"]
Jul 19 07:45:12.035719 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.035369    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" containerName="console"
Jul 19 07:45:12.035719 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.035387    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" containerName="console"
Jul 19 07:45:12.035719 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.035427    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ac2fcb-4549-40f7-a9d9-fc4857f7213b" containerName="console"
Jul 19 07:45:12.037653 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.037623    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.040142 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.040117    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 19 07:45:12.040251 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.040150    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 19 07:45:12.040386 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.040371    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 19 07:45:12.041372 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.041350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 19 07:45:12.041469 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.041361    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 19 07:45:12.041619 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.041598    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lgk5x\""
Jul 19 07:45:12.051855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.051825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"]
Jul 19 07:45:12.052407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.052385    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 19 07:45:12.096286 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096250    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-metrics-client-ca\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-federate-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096454 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd6fg\" (UniqueName: \"kubernetes.io/projected/000633b7-70d2-42c7-90ad-20c7235603a1-kube-api-access-jd6fg\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096640 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-serving-certs-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.096640 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.096584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197684 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197684 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd6fg\" (UniqueName: \"kubernetes.io/projected/000633b7-70d2-42c7-90ad-20c7235603a1-kube-api-access-jd6fg\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197714    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-serving-certs-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-metrics-client-ca\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.197957 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.197843    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-federate-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.198543 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.198514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-metrics-client-ca\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.198693 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.198529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-serving-certs-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.198978 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.198952    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.200670 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.200614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.200942 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.200920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-federate-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.201013 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.200920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.201257 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.201241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/000633b7-70d2-42c7-90ad-20c7235603a1-telemeter-client-tls\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.205638 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.205617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd6fg\" (UniqueName: \"kubernetes.io/projected/000633b7-70d2-42c7-90ad-20c7235603a1-kube-api-access-jd6fg\") pod \"telemeter-client-6f6c944bfd-n4dxs\" (UID: \"000633b7-70d2-42c7-90ad-20c7235603a1\") " pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.350715 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.350421    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"
Jul 19 07:45:12.493802 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:12.493771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs"]
Jul 19 07:45:12.497810 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:45:12.497781    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000633b7_70d2_42c7_90ad_20c7235603a1.slice/crio-191630317d3321ec8efa15e6a9ddeb712f16f8c223faee91376a10ddb53c8acd WatchSource:0}: Error finding container 191630317d3321ec8efa15e6a9ddeb712f16f8c223faee91376a10ddb53c8acd: Status 404 returned error can't find the container with id 191630317d3321ec8efa15e6a9ddeb712f16f8c223faee91376a10ddb53c8acd
Jul 19 07:45:13.251799 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:13.251763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs" event={"ID":"000633b7-70d2-42c7-90ad-20c7235603a1","Type":"ContainerStarted","Data":"191630317d3321ec8efa15e6a9ddeb712f16f8c223faee91376a10ddb53c8acd"}
Jul 19 07:45:15.263332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:15.263293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs" event={"ID":"000633b7-70d2-42c7-90ad-20c7235603a1","Type":"ContainerStarted","Data":"34fa71bf4d335452425eb7eb7ba17b73f2bd207533c60826355f0eda2c2e2d41"}
Jul 19 07:45:16.273497 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:16.273458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs" event={"ID":"000633b7-70d2-42c7-90ad-20c7235603a1","Type":"ContainerStarted","Data":"d0d82feaa9d7162e8922dde92e8fab69bd4319ba2bad35af9c1fd7e8602b04e9"}
Jul 19 07:45:16.273942 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:16.273538    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs" event={"ID":"000633b7-70d2-42c7-90ad-20c7235603a1","Type":"ContainerStarted","Data":"a66a5aff6076510552e97a214a4eacee374ad72a3d715945453bf02d40fa3774"}
Jul 19 07:45:16.293588 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:16.293535    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f6c944bfd-n4dxs" podStartSLOduration=1.388197015 podStartE2EDuration="4.293521251s" totalImagesPullingTime="2.905324236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:45:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:45:16.292291596 +0000 UTC m=+152.801889507" watchObservedRunningTime="2026-07-19 07:45:16.293521251 +0000 UTC m=+152.803119144"
Jul 19 07:45:17.004935 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.004903    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:45:17.007035 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.007020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.017419 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.017397    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:45:17.033944 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.033919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034049 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.033958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034049 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.033982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprlp\" (UniqueName: \"kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034173 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.034083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034173 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.034116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034173 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.034141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.034302 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.034194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135258 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135277    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprlp\" (UniqueName: \"kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.135456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.135409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.136185 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.136159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.136311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.136201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.136311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.136201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.136589 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.136563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.138220 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.138193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.138626 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.138601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.143994 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.143973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprlp\" (UniqueName: \"kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp\") pod \"console-7bcd594f59-w6pnb\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") " pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.316564 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.316536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:17.452855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:17.452821    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:45:17.456556 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:45:17.456528    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8b37c4_99ed_4075_a641_bca1c76de80c.slice/crio-25e2e94bdab50917e725105a1225c766c6c0b06ca1b74d7a6e3ce5f5e9f4631b WatchSource:0}: Error finding container 25e2e94bdab50917e725105a1225c766c6c0b06ca1b74d7a6e3ce5f5e9f4631b: Status 404 returned error can't find the container with id 25e2e94bdab50917e725105a1225c766c6c0b06ca1b74d7a6e3ce5f5e9f4631b
Jul 19 07:45:18.282776 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:18.282739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd594f59-w6pnb" event={"ID":"9a8b37c4-99ed-4075-a641-bca1c76de80c","Type":"ContainerStarted","Data":"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"}
Jul 19 07:45:18.283010 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:18.282805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd594f59-w6pnb" event={"ID":"9a8b37c4-99ed-4075-a641-bca1c76de80c","Type":"ContainerStarted","Data":"25e2e94bdab50917e725105a1225c766c6c0b06ca1b74d7a6e3ce5f5e9f4631b"}
Jul 19 07:45:18.299280 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:18.299232    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bcd594f59-w6pnb" podStartSLOduration=2.299218335 podStartE2EDuration="2.299218335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:45:18.297892415 +0000 UTC m=+154.807490315" watchObservedRunningTime="2026-07-19 07:45:18.299218335 +0000 UTC m=+154.808816234"
Jul 19 07:45:27.317435 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:27.317396    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:27.317904 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:27.317465    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:27.322245 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:27.322225    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:28.327757 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:28.327730    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:45:28.382902 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:28.382872    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:45:53.403805 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.403738    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-847b7b8f78-th72n" podUID="42bd4eab-8bc3-426b-8884-950cb099b0c9" containerName="console" containerID="cri-o://b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37" gracePeriod=15
Jul 19 07:45:53.642165 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.642141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:45:53.723744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723647    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723735    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723953 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723953 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723799    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723953 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723832    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723953 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723856    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvcp8\" (UniqueName: \"kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.723953 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.723886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle\") pod \"42bd4eab-8bc3-426b-8884-950cb099b0c9\" (UID: \"42bd4eab-8bc3-426b-8884-950cb099b0c9\") "
Jul 19 07:45:53.724203 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.724163    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:45:53.724203 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.724178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca" (OuterVolumeSpecName: "service-ca") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:45:53.724338 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.724314    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config" (OuterVolumeSpecName: "console-config") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:45:53.724456 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.724429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:45:53.725883 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.725858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:45:53.726031 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.726008    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8" (OuterVolumeSpecName: "kube-api-access-kvcp8") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "kube-api-access-kvcp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:45:53.726090 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.726021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "42bd4eab-8bc3-426b-8884-950cb099b0c9" (UID: "42bd4eab-8bc3-426b-8884-950cb099b0c9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:45:53.824614 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824583    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-oauth-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824614 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824609    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-oauth-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824614 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824619    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824628    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvcp8\" (UniqueName: \"kubernetes.io/projected/42bd4eab-8bc3-426b-8884-950cb099b0c9-kube-api-access-kvcp8\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824639    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-trusted-ca-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824647    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42bd4eab-8bc3-426b-8884-950cb099b0c9-console-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:53.824850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:53.824657    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42bd4eab-8bc3-426b-8884-950cb099b0c9-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:45:54.415139 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.415100    2559 generic.go:356] "Generic (PLEG): container finished" podID="42bd4eab-8bc3-426b-8884-950cb099b0c9" containerID="b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37" exitCode=0
Jul 19 07:45:54.415139 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.415145    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-847b7b8f78-th72n"
Jul 19 07:45:54.415611 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.415179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b7b8f78-th72n" event={"ID":"42bd4eab-8bc3-426b-8884-950cb099b0c9","Type":"ContainerDied","Data":"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"}
Jul 19 07:45:54.415611 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.415235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-847b7b8f78-th72n" event={"ID":"42bd4eab-8bc3-426b-8884-950cb099b0c9","Type":"ContainerDied","Data":"ed6b8e033e723127d96d6770ca01ae54ec0ab6829c84b84ee158f1dcea2f5157"}
Jul 19 07:45:54.415611 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.415262    2559 scope.go:117] "RemoveContainer" containerID="b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"
Jul 19 07:45:54.425992 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.425970    2559 scope.go:117] "RemoveContainer" containerID="b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"
Jul 19 07:45:54.426317 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:45:54.426286    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37\": container with ID starting with b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37 not found: ID does not exist" containerID="b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"
Jul 19 07:45:54.426415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.426324    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37"} err="failed to get container status \"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37\": rpc error: code = NotFound desc = could not find container \"b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37\": container with ID starting with b75ed74584c70e6eec03e6ae21386fd42a1c93fdcce11d8d9e35e26e42223a37 not found: ID does not exist"
Jul 19 07:45:54.439152 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.439126    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:45:54.444688 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.444669    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-847b7b8f78-th72n"]
Jul 19 07:45:54.459073 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:45:54.459049    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bd4eab-8bc3-426b-8884-950cb099b0c9" path="/var/lib/kubelet/pods/42bd4eab-8bc3-426b-8884-950cb099b0c9/volumes"
Jul 19 07:46:29.454936 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.454900    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:46:29.455429 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.455177    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bd4eab-8bc3-426b-8884-950cb099b0c9" containerName="console"
Jul 19 07:46:29.455429 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.455188    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bd4eab-8bc3-426b-8884-950cb099b0c9" containerName="console"
Jul 19 07:46:29.455429 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.455238    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bd4eab-8bc3-426b-8884-950cb099b0c9" containerName="console"
Jul 19 07:46:29.457865 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.457848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.467348 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.467325    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:46:29.482859 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.482971 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482867    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6t28\" (UniqueName: \"kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.482971 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.482971 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.482971 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.483153 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.482998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.483153 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.483015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584296 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584452 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6t28\" (UniqueName: \"kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584452 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584452 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584574 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584574 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.584678 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.584596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.585212 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.585189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.585312 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.585269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.585312 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.585290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.585507 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.585490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.587237 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.587220    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.587352 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.587332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.591452 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.591433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6t28\" (UniqueName: \"kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28\") pod \"console-669d468d7c-296mw\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") " pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.767337 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.767255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:29.893332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:29.893168    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:46:29.896000 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:46:29.895973    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43df65af_932f_4565_9009_86a5af68256a.slice/crio-cb0af952edcf4fd30f0360087b5b6f7c58ef77b83f640b1c3d9a55c772aab635 WatchSource:0}: Error finding container cb0af952edcf4fd30f0360087b5b6f7c58ef77b83f640b1c3d9a55c772aab635: Status 404 returned error can't find the container with id cb0af952edcf4fd30f0360087b5b6f7c58ef77b83f640b1c3d9a55c772aab635
Jul 19 07:46:30.536217 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:30.536185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d468d7c-296mw" event={"ID":"43df65af-932f-4565-9009-86a5af68256a","Type":"ContainerStarted","Data":"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"}
Jul 19 07:46:30.536656 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:30.536231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d468d7c-296mw" event={"ID":"43df65af-932f-4565-9009-86a5af68256a","Type":"ContainerStarted","Data":"cb0af952edcf4fd30f0360087b5b6f7c58ef77b83f640b1c3d9a55c772aab635"}
Jul 19 07:46:30.553355 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:30.553291    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-669d468d7c-296mw" podStartSLOduration=1.553269789 podStartE2EDuration="1.553269789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:46:30.553046915 +0000 UTC m=+227.062644815" watchObservedRunningTime="2026-07-19 07:46:30.553269789 +0000 UTC m=+227.062867690"
Jul 19 07:46:39.767887 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:39.767849    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:39.768344 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:39.767904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:39.773152 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:39.773127    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:40.575617 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:40.575587    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:46:40.616530 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:46:40.616485    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:47:05.637025 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.636932    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bcd594f59-w6pnb" podUID="9a8b37c4-99ed-4075-a641-bca1c76de80c" containerName="console" containerID="cri-o://9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f" gracePeriod=15
Jul 19 07:47:05.879254 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.879231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:47:05.966419 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966348    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vprlp\" (UniqueName: \"kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966419 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966385    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966419 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966643 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966643 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966449    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966643 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966483    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966643 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert\") pod \"9a8b37c4-99ed-4075-a641-bca1c76de80c\" (UID: \"9a8b37c4-99ed-4075-a641-bca1c76de80c\") "
Jul 19 07:47:05.966899 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966864    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:47:05.966899 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config" (OuterVolumeSpecName: "console-config") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:47:05.967065 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966913    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:47:05.967065 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.966960    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:47:05.968744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.968694    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp" (OuterVolumeSpecName: "kube-api-access-vprlp") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "kube-api-access-vprlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:47:05.968744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.968732    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:47:05.968744 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:05.968714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a8b37c4-99ed-4075-a641-bca1c76de80c" (UID: "9a8b37c4-99ed-4075-a641-bca1c76de80c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:47:06.067762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067725    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067757    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprlp\" (UniqueName: \"kubernetes.io/projected/9a8b37c4-99ed-4075-a641-bca1c76de80c-kube-api-access-vprlp\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067767    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-oauth-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067931 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067776    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067931 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067785    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a8b37c4-99ed-4075-a641-bca1c76de80c-console-oauth-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067931 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067793    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-trusted-ca-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.067931 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.067800    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8b37c4-99ed-4075-a641-bca1c76de80c-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:06.669491 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.669453    2559 generic.go:356] "Generic (PLEG): container finished" podID="9a8b37c4-99ed-4075-a641-bca1c76de80c" containerID="9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f" exitCode=0
Jul 19 07:47:06.669491 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.669494    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd594f59-w6pnb" event={"ID":"9a8b37c4-99ed-4075-a641-bca1c76de80c","Type":"ContainerDied","Data":"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"}
Jul 19 07:47:06.670020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.669505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd594f59-w6pnb"
Jul 19 07:47:06.670020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.669532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd594f59-w6pnb" event={"ID":"9a8b37c4-99ed-4075-a641-bca1c76de80c","Type":"ContainerDied","Data":"25e2e94bdab50917e725105a1225c766c6c0b06ca1b74d7a6e3ce5f5e9f4631b"}
Jul 19 07:47:06.670020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.669556    2559 scope.go:117] "RemoveContainer" containerID="9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"
Jul 19 07:47:06.677059 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.677027    2559 scope.go:117] "RemoveContainer" containerID="9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"
Jul 19 07:47:06.677340 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:47:06.677323    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f\": container with ID starting with 9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f not found: ID does not exist" containerID="9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"
Jul 19 07:47:06.677391 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.677349    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f"} err="failed to get container status \"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f\": rpc error: code = NotFound desc = could not find container \"9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f\": container with ID starting with 9dffcca4b6d759491a258058fd1880d3a43b0b5f9b974bfcef98860bb4ece14f not found: ID does not exist"
Jul 19 07:47:06.683734 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.683716    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:47:06.687043 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:06.687025    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcd594f59-w6pnb"]
Jul 19 07:47:08.459358 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:08.459325    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8b37c4-99ed-4075-a641-bca1c76de80c" path="/var/lib/kubelet/pods/9a8b37c4-99ed-4075-a641-bca1c76de80c/volumes"
Jul 19 07:47:33.549259 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.549227    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"]
Jul 19 07:47:33.549722 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.549516    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8b37c4-99ed-4075-a641-bca1c76de80c" containerName="console"
Jul 19 07:47:33.549722 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.549528    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8b37c4-99ed-4075-a641-bca1c76de80c" containerName="console"
Jul 19 07:47:33.549722 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.549579    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8b37c4-99ed-4075-a641-bca1c76de80c" containerName="console"
Jul 19 07:47:33.554039 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.554011    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.556555 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.556535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6vmfl\""
Jul 19 07:47:33.556657 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.556561    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 19 07:47:33.556657 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.556562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 19 07:47:33.560695 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.560662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"]
Jul 19 07:47:33.560844 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.560738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.562003 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.561892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.562003 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.561962    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64p8\" (UniqueName: \"kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.662943 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.662911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.663117 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.662955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.663117 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.662987    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64p8\" (UniqueName: \"kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.663333 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.663310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.663398 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.663361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.670824 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.670797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64p8\" (UniqueName: \"kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.869779 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.869748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:33.992695 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:33.992669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"]
Jul 19 07:47:33.995628 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:47:33.995601    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0cac87b_ef6f_4e21_bad1_6fcc20a0563a.slice/crio-05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751 WatchSource:0}: Error finding container 05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751: Status 404 returned error can't find the container with id 05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751
Jul 19 07:47:34.770196 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:34.770156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8" event={"ID":"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a","Type":"ContainerStarted","Data":"05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751"}
Jul 19 07:47:40.794374 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:40.794281    2559 generic.go:356] "Generic (PLEG): container finished" podID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerID="428c1cebc654e7dce1d7c90f5f362e1f4f852847d5acb62e4fe4ba5cfafcf2c9" exitCode=0
Jul 19 07:47:40.794374 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:40.794351    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8" event={"ID":"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a","Type":"ContainerDied","Data":"428c1cebc654e7dce1d7c90f5f362e1f4f852847d5acb62e4fe4ba5cfafcf2c9"}
Jul 19 07:47:43.808154 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:43.808118    2559 generic.go:356] "Generic (PLEG): container finished" podID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerID="8a9969a9b258b0de71faa4e69d486fc0d07b8d7d71effab2186be33857173c5a" exitCode=0
Jul 19 07:47:43.808541 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:43.808169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8" event={"ID":"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a","Type":"ContainerDied","Data":"8a9969a9b258b0de71faa4e69d486fc0d07b8d7d71effab2186be33857173c5a"}
Jul 19 07:47:44.345009 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:44.344980    2559 kubelet.go:1648] "Image garbage collection succeeded"
Jul 19 07:47:50.837341 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:50.837310    2559 generic.go:356] "Generic (PLEG): container finished" podID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerID="89afd62bcef603845b37b4b14d4ef47bf879c975883699cab50f372043d663d8" exitCode=0
Jul 19 07:47:50.837762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:50.837364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8" event={"ID":"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a","Type":"ContainerDied","Data":"89afd62bcef603845b37b4b14d4ef47bf879c975883699cab50f372043d663d8"}
Jul 19 07:47:51.967250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:51.967224    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:52.005060 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.005033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w64p8\" (UniqueName: \"kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8\") pod \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") "
Jul 19 07:47:52.005180 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.005085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util\") pod \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") "
Jul 19 07:47:52.005180 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.005134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle\") pod \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\" (UID: \"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a\") "
Jul 19 07:47:52.005804 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.005778    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle" (OuterVolumeSpecName: "bundle") pod "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" (UID: "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:47:52.007079 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.007050    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8" (OuterVolumeSpecName: "kube-api-access-w64p8") pod "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" (UID: "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a"). InnerVolumeSpecName "kube-api-access-w64p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:47:52.009205 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.009187    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util" (OuterVolumeSpecName: "util") pod "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" (UID: "f0cac87b-ef6f-4e21-bad1-6fcc20a0563a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:47:52.106002 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.105935    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:52.106002 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.105968    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w64p8\" (UniqueName: \"kubernetes.io/projected/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-kube-api-access-w64p8\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:52.106002 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.105977    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0cac87b-ef6f-4e21-bad1-6fcc20a0563a-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:47:52.846183 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.846159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8"
Jul 19 07:47:52.846183 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.846160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408xlm8" event={"ID":"f0cac87b-ef6f-4e21-bad1-6fcc20a0563a","Type":"ContainerDied","Data":"05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751"}
Jul 19 07:47:52.846361 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:52.846209    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f09ba648fc9a2bf8f18c8acf13020f240e95956adf68ccbd4a2be2917df751"
Jul 19 07:47:56.509562 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509529    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"]
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509889    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="extract"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509903    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="extract"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509912    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="pull"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509918    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="pull"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509927    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="util"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509933    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="util"
Jul 19 07:47:56.510129 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.509990    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0cac87b-ef6f-4e21-bad1-6fcc20a0563a" containerName="extract"
Jul 19 07:47:56.513120 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.513099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.515471 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.515451    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 19 07:47:56.515548 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.515485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-krh47\""
Jul 19 07:47:56.515548 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.515483    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 19 07:47:56.524128 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.524108    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"]
Jul 19 07:47:56.638588 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.638552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.638775 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.638634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zmh\" (UniqueName: \"kubernetes.io/projected/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-kube-api-access-z6zmh\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.739822 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.739792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zmh\" (UniqueName: \"kubernetes.io/projected/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-kube-api-access-z6zmh\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.739903 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.739860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.740288 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.740268    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.750648 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.750620    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zmh\" (UniqueName: \"kubernetes.io/projected/03ef5099-0c3a-4195-bc6b-0ef8cf0c346c-kube-api-access-z6zmh\") pod \"cert-manager-operator-controller-manager-55c444bf86-qqhd5\" (UID: \"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.821662 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.821640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"
Jul 19 07:47:56.972341 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.972317    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5"]
Jul 19 07:47:56.974734 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:47:56.974691    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ef5099_0c3a_4195_bc6b_0ef8cf0c346c.slice/crio-bf558b55962540c304108e0a25cae1901650d2d937624868b5b16e6853de0f08 WatchSource:0}: Error finding container bf558b55962540c304108e0a25cae1901650d2d937624868b5b16e6853de0f08: Status 404 returned error can't find the container with id bf558b55962540c304108e0a25cae1901650d2d937624868b5b16e6853de0f08
Jul 19 07:47:56.978005 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:56.977990    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 07:47:57.866794 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:57.866753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5" event={"ID":"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c","Type":"ContainerStarted","Data":"bf558b55962540c304108e0a25cae1901650d2d937624868b5b16e6853de0f08"}
Jul 19 07:47:58.877446 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:58.877408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5" event={"ID":"03ef5099-0c3a-4195-bc6b-0ef8cf0c346c","Type":"ContainerStarted","Data":"e43a2b3e623bbf23a4a53fd907af630a77061dddbf0d99440416ac306e8ee975"}
Jul 19 07:47:58.897565 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:47:58.897516    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qqhd5" podStartSLOduration=1.432148198 podStartE2EDuration="2.897496598s" totalImagesPullingTime="1.4653484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:47:58.896228703 +0000 UTC m=+315.405826615" watchObservedRunningTime="2026-07-19 07:47:58.897496598 +0000 UTC m=+315.407094490"
Jul 19 07:48:00.143447 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.143410    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"]
Jul 19 07:48:00.145548 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.145531    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.148068 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.148048    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 19 07:48:00.148896 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.148882    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6vmfl\""
Jul 19 07:48:00.148962 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.148892    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 19 07:48:00.155590 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.155568    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"]
Jul 19 07:48:00.269469 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.269427    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.269635 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.269489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.269635 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.269534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqht\" (UniqueName: \"kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.370402 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.370371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqht\" (UniqueName: \"kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.370566 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.370417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.370566 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.370482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.370891 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.370872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.370933 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.370904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.379058 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.379029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqht\" (UniqueName: \"kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.454588 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.454518    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:00.576901 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.576870    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"]
Jul 19 07:48:00.579934 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:00.579911    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4b6ce7_2ead_4ebc_8891_e1d6962991c2.slice/crio-77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f WatchSource:0}: Error finding container 77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f: Status 404 returned error can't find the container with id 77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f
Jul 19 07:48:00.887170 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.887142    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerID="e0c5f02b6cf89a8293cc89e8231dbd9831f26711bbbfea9461f5c749deb1b4df" exitCode=0
Jul 19 07:48:00.887338 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.887232    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq" event={"ID":"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2","Type":"ContainerDied","Data":"e0c5f02b6cf89a8293cc89e8231dbd9831f26711bbbfea9461f5c749deb1b4df"}
Jul 19 07:48:00.887338 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:00.887285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq" event={"ID":"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2","Type":"ContainerStarted","Data":"77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f"}
Jul 19 07:48:02.586552 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.586520    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"]
Jul 19 07:48:02.593616 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.593576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.597116 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.597027    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 19 07:48:02.597317 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.597300    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 19 07:48:02.598119 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.597843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6gbpg\""
Jul 19 07:48:02.599645 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.599624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"]
Jul 19 07:48:02.688928 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.688896    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9t4\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-kube-api-access-vm9t4\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.689096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.688958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.789955 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.789934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.790040 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.790020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9t4\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-kube-api-access-vm9t4\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.796849 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.796827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.797167 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.797148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9t4\" (UniqueName: \"kubernetes.io/projected/3943f676-69ae-4c2e-b205-22f5e1b89118-kube-api-access-vm9t4\") pod \"cert-manager-cainjector-559cdd8c87-c94dl\" (UID: \"3943f676-69ae-4c2e-b205-22f5e1b89118\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:02.903990 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.903955    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerID="d6f57479a81c32fe06574338edbf2605102b8cae1ba23d2e91bb71a413477a54" exitCode=0
Jul 19 07:48:02.904149 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.904042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq" event={"ID":"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2","Type":"ContainerDied","Data":"d6f57479a81c32fe06574338edbf2605102b8cae1ba23d2e91bb71a413477a54"}
Jul 19 07:48:02.915650 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:02.915632    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"
Jul 19 07:48:03.045060 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:03.045031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-c94dl"]
Jul 19 07:48:03.059499 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:03.059468    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3943f676_69ae_4c2e_b205_22f5e1b89118.slice/crio-beb090b09413a3ff8af4aee56bd2fada756b2ee812eb85aa3ca8a368e6375b5a WatchSource:0}: Error finding container beb090b09413a3ff8af4aee56bd2fada756b2ee812eb85aa3ca8a368e6375b5a: Status 404 returned error can't find the container with id beb090b09413a3ff8af4aee56bd2fada756b2ee812eb85aa3ca8a368e6375b5a
Jul 19 07:48:03.908909 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:03.908874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl" event={"ID":"3943f676-69ae-4c2e-b205-22f5e1b89118","Type":"ContainerStarted","Data":"beb090b09413a3ff8af4aee56bd2fada756b2ee812eb85aa3ca8a368e6375b5a"}
Jul 19 07:48:03.910820 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:03.910795    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerID="5c02a134e9a3767879db7b2ae1a052f19a23aeb0f7e19c65d4c223765966d965" exitCode=0
Jul 19 07:48:03.910949 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:03.910878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq" event={"ID":"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2","Type":"ContainerDied","Data":"5c02a134e9a3767879db7b2ae1a052f19a23aeb0f7e19c65d4c223765966d965"}
Jul 19 07:48:05.063962 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.063934    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:05.110339 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.110308    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle\") pod \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") "
Jul 19 07:48:05.110510 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.110363    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgqht\" (UniqueName: \"kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht\") pod \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") "
Jul 19 07:48:05.110510 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.110448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util\") pod \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\" (UID: \"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2\") "
Jul 19 07:48:05.110848 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.110816    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle" (OuterVolumeSpecName: "bundle") pod "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" (UID: "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:48:05.112717 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.112680    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht" (OuterVolumeSpecName: "kube-api-access-tgqht") pod "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" (UID: "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2"). InnerVolumeSpecName "kube-api-access-tgqht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:48:05.118745 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.118717    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util" (OuterVolumeSpecName: "util") pod "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" (UID: "7c4b6ce7-2ead-4ebc-8891-e1d6962991c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:48:05.211794 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.211717    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:05.211794 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.211742    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqht\" (UniqueName: \"kubernetes.io/projected/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-kube-api-access-tgqht\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:05.211794 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.211753    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c4b6ce7-2ead-4ebc-8891-e1d6962991c2-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:05.922246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.922213    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq" event={"ID":"7c4b6ce7-2ead-4ebc-8891-e1d6962991c2","Type":"ContainerDied","Data":"77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f"}
Jul 19 07:48:05.922436 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.922265    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c0dad8917b665ef26a6341c2b22d9fbb314160fb6dec52121ca4941c293c3f"
Jul 19 07:48:05.922436 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:05.922222    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcwdgq"
Jul 19 07:48:06.929822 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:06.929782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl" event={"ID":"3943f676-69ae-4c2e-b205-22f5e1b89118","Type":"ContainerStarted","Data":"04f2eae4647194753ef477ba3f262f9bf216cd2ae94def8995881164585fc53a"}
Jul 19 07:48:06.944173 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:06.944120    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-c94dl" podStartSLOduration=1.862683152 podStartE2EDuration="4.944102695s" totalImagesPullingTime="3.081419543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:06.94366086 +0000 UTC m=+323.453258760" watchObservedRunningTime="2026-07-19 07:48:06.944102695 +0000 UTC m=+323.453700595"
Jul 19 07:48:08.096275 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096238    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cmr22"]
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096509    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="extract"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096520    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="extract"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096540    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="pull"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096545    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="pull"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096553    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="util"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096558    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="util"
Jul 19 07:48:08.096743 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.096599    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4b6ce7-2ead-4ebc-8891-e1d6962991c2" containerName="extract"
Jul 19 07:48:08.099448 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.099429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.101897 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.101877    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7xtsz\""
Jul 19 07:48:08.108725 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.108680    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cmr22"]
Jul 19 07:48:08.235740 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.235685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4qc\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-kube-api-access-rc4qc\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.235881 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.235757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.336337 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.336309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4qc\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-kube-api-access-rc4qc\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.336510 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.336365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.347532 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.347472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.347681 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.347661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4qc\" (UniqueName: \"kubernetes.io/projected/cd020b37-5015-4e6f-98a7-0f068cd6751a-kube-api-access-rc4qc\") pod \"cert-manager-webhook-6486ffd997-cmr22\" (UID: \"cd020b37-5015-4e6f-98a7-0f068cd6751a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.409510 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.409473    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.534037 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.534013    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-cmr22"]
Jul 19 07:48:08.536375 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:08.536342    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd020b37_5015_4e6f_98a7_0f068cd6751a.slice/crio-f82e8ef83c06ab802c27a7859d0109dae75b3cfd0312c80ead7131eaa26d6d38 WatchSource:0}: Error finding container f82e8ef83c06ab802c27a7859d0109dae75b3cfd0312c80ead7131eaa26d6d38: Status 404 returned error can't find the container with id f82e8ef83c06ab802c27a7859d0109dae75b3cfd0312c80ead7131eaa26d6d38
Jul 19 07:48:08.940545 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.940509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22" event={"ID":"cd020b37-5015-4e6f-98a7-0f068cd6751a","Type":"ContainerStarted","Data":"6f2758479b1f868642df5bb5d70e9c7da86134f1fb0d3f6b4f42b4b2af1b2a75"}
Jul 19 07:48:08.940545 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.940549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22" event={"ID":"cd020b37-5015-4e6f-98a7-0f068cd6751a","Type":"ContainerStarted","Data":"f82e8ef83c06ab802c27a7859d0109dae75b3cfd0312c80ead7131eaa26d6d38"}
Jul 19 07:48:08.940816 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.940638    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:08.958627 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:08.958582    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22" podStartSLOduration=0.958565422 podStartE2EDuration="958.565422ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:08.957570854 +0000 UTC m=+325.467168753" watchObservedRunningTime="2026-07-19 07:48:08.958565422 +0000 UTC m=+325.468163320"
Jul 19 07:48:11.163162 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.163130    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-km9rq"]
Jul 19 07:48:11.166199 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.166182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.168470 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.168450    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zb54h\""
Jul 19 07:48:11.175074 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.175053    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-km9rq"]
Jul 19 07:48:11.260790 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.260763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-bound-sa-token\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.260907 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.260808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bh9\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-kube-api-access-h4bh9\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.361693 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.361653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bh9\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-kube-api-access-h4bh9\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.361889 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.361748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-bound-sa-token\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.374956 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.374930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bh9\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-kube-api-access-h4bh9\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.374956 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.374942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c727cad-95e3-4604-b04e-0b9122d62cdf-bound-sa-token\") pod \"cert-manager-55b4d954d5-km9rq\" (UID: \"2c727cad-95e3-4604-b04e-0b9122d62cdf\") " pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.475933 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.475854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-km9rq"
Jul 19 07:48:11.596390 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.596248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-km9rq"]
Jul 19 07:48:11.599031 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:11.598993    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c727cad_95e3_4604_b04e_0b9122d62cdf.slice/crio-fb8d1e89db56dd7704d5b9ac113f21fef7ad41eb68d828dfa2c4d0165914dfce WatchSource:0}: Error finding container fb8d1e89db56dd7704d5b9ac113f21fef7ad41eb68d828dfa2c4d0165914dfce: Status 404 returned error can't find the container with id fb8d1e89db56dd7704d5b9ac113f21fef7ad41eb68d828dfa2c4d0165914dfce
Jul 19 07:48:11.956850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.956817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-km9rq" event={"ID":"2c727cad-95e3-4604-b04e-0b9122d62cdf","Type":"ContainerStarted","Data":"45c35ea6fbe1a0a453c12d2067fbffb27a236cd323554339e95bb4516a4d2cc4"}
Jul 19 07:48:11.956850 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.956856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-km9rq" event={"ID":"2c727cad-95e3-4604-b04e-0b9122d62cdf","Type":"ContainerStarted","Data":"fb8d1e89db56dd7704d5b9ac113f21fef7ad41eb68d828dfa2c4d0165914dfce"}
Jul 19 07:48:11.972317 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:11.972272    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-km9rq" podStartSLOduration=0.972255608 podStartE2EDuration="972.255608ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:11.970940284 +0000 UTC m=+328.480538185" watchObservedRunningTime="2026-07-19 07:48:11.972255608 +0000 UTC m=+328.481853506"
Jul 19 07:48:13.179268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.179235    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"]
Jul 19 07:48:13.182431 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.182415    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.184894 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.184868    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 19 07:48:13.185021 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.184868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q5hc7\""
Jul 19 07:48:13.185871 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.185850    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 19 07:48:13.189940 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.189920    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"]
Jul 19 07:48:13.277691 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.277659    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9e950e-53df-475c-8e95-7aba1ee767eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.277844 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.277696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbz6f\" (UniqueName: \"kubernetes.io/projected/7f9e950e-53df-475c-8e95-7aba1ee767eb-kube-api-access-tbz6f\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.378248 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.378217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9e950e-53df-475c-8e95-7aba1ee767eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.378414 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.378255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbz6f\" (UniqueName: \"kubernetes.io/projected/7f9e950e-53df-475c-8e95-7aba1ee767eb-kube-api-access-tbz6f\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.378579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.378558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f9e950e-53df-475c-8e95-7aba1ee767eb-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.386160 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.386138    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbz6f\" (UniqueName: \"kubernetes.io/projected/7f9e950e-53df-475c-8e95-7aba1ee767eb-kube-api-access-tbz6f\") pod \"openshift-lws-operator-fd8ccff4c-4knbx\" (UID: \"7f9e950e-53df-475c-8e95-7aba1ee767eb\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.492677 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.492592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"
Jul 19 07:48:13.617434 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.617399    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx"]
Jul 19 07:48:13.620095 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:13.620063    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f9e950e_53df_475c_8e95_7aba1ee767eb.slice/crio-5b482e54a671660610061cdf08bdcb0660d2b0f3895938c3e817c8b9ec6f8925 WatchSource:0}: Error finding container 5b482e54a671660610061cdf08bdcb0660d2b0f3895938c3e817c8b9ec6f8925: Status 404 returned error can't find the container with id 5b482e54a671660610061cdf08bdcb0660d2b0f3895938c3e817c8b9ec6f8925
Jul 19 07:48:13.967756 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:13.967723    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx" event={"ID":"7f9e950e-53df-475c-8e95-7aba1ee767eb","Type":"ContainerStarted","Data":"5b482e54a671660610061cdf08bdcb0660d2b0f3895938c3e817c8b9ec6f8925"}
Jul 19 07:48:14.947855 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:14.947826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-cmr22"
Jul 19 07:48:15.980492 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:15.980458    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx" event={"ID":"7f9e950e-53df-475c-8e95-7aba1ee767eb","Type":"ContainerStarted","Data":"b0ea302823d759e0a6fcf062b438c7319d80d5bc04c15acc43c786bc5e5b624d"}
Jul 19 07:48:15.995763 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:15.995719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4knbx" podStartSLOduration=1.380367753 podStartE2EDuration="2.995684903s" totalImagesPullingTime="1.61531715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:15.99448785 +0000 UTC m=+332.504085748" watchObservedRunningTime="2026-07-19 07:48:15.995684903 +0000 UTC m=+332.505282802"
Jul 19 07:48:20.980815 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.980779    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"]
Jul 19 07:48:20.984215 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.984195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:20.986801 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.986776    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6vmfl\""
Jul 19 07:48:20.986926 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.986820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 19 07:48:20.987785 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.987770    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 19 07:48:20.992199 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:20.992175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"]
Jul 19 07:48:21.148199 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.148167    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.148379 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.148219    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.148379 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.148256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8zg\" (UniqueName: \"kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.249721 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.249618    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.249721 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.249682    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8zg\" (UniqueName: \"kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.249904 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.249728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.250091 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.250074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.250127 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.250088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.257680 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.257652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8zg\" (UniqueName: \"kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.293652 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.293622    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:21.422540 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:21.422513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"]
Jul 19 07:48:21.424517 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:21.424487    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbca4d5d_96da_4805_93da_6478fb4ff5ce.slice/crio-d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324 WatchSource:0}: Error finding container d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324: Status 404 returned error can't find the container with id d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324
Jul 19 07:48:22.008108 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:22.008069    2559 generic.go:356] "Generic (PLEG): container finished" podID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerID="751c6fbffc25a0ef7b23f86102b97b90a658b8c8a6681cf536b5287f6d1f577c" exitCode=0
Jul 19 07:48:22.008632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:22.008149    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx" event={"ID":"bbca4d5d-96da-4805-93da-6478fb4ff5ce","Type":"ContainerDied","Data":"751c6fbffc25a0ef7b23f86102b97b90a658b8c8a6681cf536b5287f6d1f577c"}
Jul 19 07:48:22.008632 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:22.008204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx" event={"ID":"bbca4d5d-96da-4805-93da-6478fb4ff5ce","Type":"ContainerStarted","Data":"d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324"}
Jul 19 07:48:23.014758 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:23.014727    2559 generic.go:356] "Generic (PLEG): container finished" podID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerID="a2655d7a5630b93f3b1de040a45abef826757148793d16f3aab73248bd63632a" exitCode=0
Jul 19 07:48:23.015234 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:23.014813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx" event={"ID":"bbca4d5d-96da-4805-93da-6478fb4ff5ce","Type":"ContainerDied","Data":"a2655d7a5630b93f3b1de040a45abef826757148793d16f3aab73248bd63632a"}
Jul 19 07:48:24.021655 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:24.021617    2559 generic.go:356] "Generic (PLEG): container finished" podID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerID="d01a4d7088597caa547e9341d4e32baed04d279bb3a83af41da30e5c9b84d923" exitCode=0
Jul 19 07:48:24.022061 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:24.021671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx" event={"ID":"bbca4d5d-96da-4805-93da-6478fb4ff5ce","Type":"ContainerDied","Data":"d01a4d7088597caa547e9341d4e32baed04d279bb3a83af41da30e5c9b84d923"}
Jul 19 07:48:25.177145 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.177116    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:25.285294 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.285212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8zg\" (UniqueName: \"kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg\") pod \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") "
Jul 19 07:48:25.285294 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.285256    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util\") pod \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") "
Jul 19 07:48:25.285471 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.285310    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle\") pod \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\" (UID: \"bbca4d5d-96da-4805-93da-6478fb4ff5ce\") "
Jul 19 07:48:25.286110 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.286080    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle" (OuterVolumeSpecName: "bundle") pod "bbca4d5d-96da-4805-93da-6478fb4ff5ce" (UID: "bbca4d5d-96da-4805-93da-6478fb4ff5ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:48:25.287279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.287235    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg" (OuterVolumeSpecName: "kube-api-access-qx8zg") pod "bbca4d5d-96da-4805-93da-6478fb4ff5ce" (UID: "bbca4d5d-96da-4805-93da-6478fb4ff5ce"). InnerVolumeSpecName "kube-api-access-qx8zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:48:25.290546 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.290508    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util" (OuterVolumeSpecName: "util") pod "bbca4d5d-96da-4805-93da-6478fb4ff5ce" (UID: "bbca4d5d-96da-4805-93da-6478fb4ff5ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:48:25.385915 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.385880    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:25.385915 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.385908    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8zg\" (UniqueName: \"kubernetes.io/projected/bbca4d5d-96da-4805-93da-6478fb4ff5ce-kube-api-access-qx8zg\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:25.385915 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:25.385919    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbca4d5d-96da-4805-93da-6478fb4ff5ce-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:48:26.035161 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:26.035114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx" event={"ID":"bbca4d5d-96da-4805-93da-6478fb4ff5ce","Type":"ContainerDied","Data":"d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324"}
Jul 19 07:48:26.035161 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:26.035142    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af698rx"
Jul 19 07:48:26.035375 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:26.035172    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d167ce21ead0dd72f76fca3bb4d9c90fc285ea968551f3b638c2250a2bb8f324"
Jul 19 07:48:34.324043 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324005    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"]
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324355    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="pull"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324367    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="pull"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324376    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="util"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324381    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="util"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324392    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="extract"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324398    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="extract"
Jul 19 07:48:34.324483 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.324445    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbca4d5d-96da-4805-93da-6478fb4ff5ce" containerName="extract"
Jul 19 07:48:34.330022 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.329999    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.332731 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.332690    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 19 07:48:34.333722 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.333681    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8c278\""
Jul 19 07:48:34.334256 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.334233    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 19 07:48:34.334353 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.334320    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 19 07:48:34.335237 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.335222    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 19 07:48:34.356840 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.356812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.356932 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.356855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.356932 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.356878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qrd\" (UniqueName: \"kubernetes.io/projected/a0057c50-2ac0-481a-96bd-f91c8a81cdee-kube-api-access-q7qrd\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.357000 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.356931    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.357000 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.356972    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.369606 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.369584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"]
Jul 19 07:48:34.457372 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.457570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.457570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qrd\" (UniqueName: \"kubernetes.io/projected/a0057c50-2ac0-481a-96bd-f91c8a81cdee-kube-api-access-q7qrd\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.457570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.457570 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.457877 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.457848    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-manifests\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.459924 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.459892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0057c50-2ac0-481a-96bd-f91c8a81cdee-tmp\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.460163 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.460139    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.460356 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.460335    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a0057c50-2ac0-481a-96bd-f91c8a81cdee-webhook-cert\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.469358 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.469331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qrd\" (UniqueName: \"kubernetes.io/projected/a0057c50-2ac0-481a-96bd-f91c8a81cdee-kube-api-access-q7qrd\") pod \"opendatahub-operator-controller-manager-789467ff8c-mz9r9\" (UID: \"a0057c50-2ac0-481a-96bd-f91c8a81cdee\") " pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.640515 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.640423    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:34.799910 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.799872    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"]
Jul 19 07:48:34.805078 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.805055    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.808155 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.808139    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tkwth\""
Jul 19 07:48:34.808934 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.808917    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 19 07:48:34.809011 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.808937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 19 07:48:34.809011 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.808921    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 19 07:48:34.818496 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.818478    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"]
Jul 19 07:48:34.821726 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:34.821681    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0057c50_2ac0_481a_96bd_f91c8a81cdee.slice/crio-9774c37e05561e7285d3112249321aae5813720d358f76a207fea140e1b3cb19 WatchSource:0}: Error finding container 9774c37e05561e7285d3112249321aae5813720d358f76a207fea140e1b3cb19: Status 404 returned error can't find the container with id 9774c37e05561e7285d3112249321aae5813720d358f76a207fea140e1b3cb19
Jul 19 07:48:34.825264 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.825244    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"]
Jul 19 07:48:34.860969 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.860943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f1cb580-bb55-48cd-9be3-6120007a8630-manager-config\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.861069 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.860979    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bx8\" (UniqueName: \"kubernetes.io/projected/8f1cb580-bb55-48cd-9be3-6120007a8630-kube-api-access-g9bx8\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.861069 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.861017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.861069 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.861052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-metrics-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.961526 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.961436    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-metrics-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.961526 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.961514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f1cb580-bb55-48cd-9be3-6120007a8630-manager-config\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.961735 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.961551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bx8\" (UniqueName: \"kubernetes.io/projected/8f1cb580-bb55-48cd-9be3-6120007a8630-kube-api-access-g9bx8\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.961735 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.961583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.962251 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.962230    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8f1cb580-bb55-48cd-9be3-6120007a8630-manager-config\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.964024 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.964002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-metrics-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.964113 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.964070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f1cb580-bb55-48cd-9be3-6120007a8630-cert\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:34.969873 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:34.969853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bx8\" (UniqueName: \"kubernetes.io/projected/8f1cb580-bb55-48cd-9be3-6120007a8630-kube-api-access-g9bx8\") pod \"lws-controller-manager-747d9ff956-clwgr\" (UID: \"8f1cb580-bb55-48cd-9be3-6120007a8630\") " pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:35.079091 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:35.079053    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9" event={"ID":"a0057c50-2ac0-481a-96bd-f91c8a81cdee","Type":"ContainerStarted","Data":"9774c37e05561e7285d3112249321aae5813720d358f76a207fea140e1b3cb19"}
Jul 19 07:48:35.114934 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:35.114906    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:35.251884 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:35.251714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"]
Jul 19 07:48:35.262473 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:48:35.262447    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1cb580_bb55_48cd_9be3_6120007a8630.slice/crio-b7d88741dc0ce704c6d10a099ba2c1abe6d69652f377d3b5544fda9502229590 WatchSource:0}: Error finding container b7d88741dc0ce704c6d10a099ba2c1abe6d69652f377d3b5544fda9502229590: Status 404 returned error can't find the container with id b7d88741dc0ce704c6d10a099ba2c1abe6d69652f377d3b5544fda9502229590
Jul 19 07:48:36.088529 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:36.088484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr" event={"ID":"8f1cb580-bb55-48cd-9be3-6120007a8630","Type":"ContainerStarted","Data":"b7d88741dc0ce704c6d10a099ba2c1abe6d69652f377d3b5544fda9502229590"}
Jul 19 07:48:39.105774 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:39.105735    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9" event={"ID":"a0057c50-2ac0-481a-96bd-f91c8a81cdee","Type":"ContainerStarted","Data":"5ce91b01132074d31870079e6dd8f1ba9fad4539519e526fe8191b4499e7beff"}
Jul 19 07:48:39.107236 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:39.107211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr" event={"ID":"8f1cb580-bb55-48cd-9be3-6120007a8630","Type":"ContainerStarted","Data":"0e26a21bd434986a0df91560bd170337632812ddda6376d9ab8b20858697bc74"}
Jul 19 07:48:39.107352 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:39.107270    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:39.143365 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:39.143317    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr" podStartSLOduration=2.192582633 podStartE2EDuration="5.143302032s" totalImagesPullingTime="2.950719399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:39.142059881 +0000 UTC m=+355.651657782" watchObservedRunningTime="2026-07-19 07:48:39.143302032 +0000 UTC m=+355.652899930"
Jul 19 07:48:40.112489 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:40.112453    2559 generic.go:356] "Generic (PLEG): container finished" podID="a0057c50-2ac0-481a-96bd-f91c8a81cdee" containerID="5ce91b01132074d31870079e6dd8f1ba9fad4539519e526fe8191b4499e7beff" exitCode=0
Jul 19 07:48:40.112904 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:40.112541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9" event={"ID":"a0057c50-2ac0-481a-96bd-f91c8a81cdee","Type":"ContainerDied","Data":"5ce91b01132074d31870079e6dd8f1ba9fad4539519e526fe8191b4499e7beff"}
Jul 19 07:48:41.119090 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:41.119050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9" event={"ID":"a0057c50-2ac0-481a-96bd-f91c8a81cdee","Type":"ContainerStarted","Data":"e0f839018bbe2b9c78038baef363b891b94b766b7e4edfe087a96edce6e7e8cd"}
Jul 19 07:48:41.119515 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:41.119170    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:48:41.140112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:41.140058    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9" podStartSLOduration=2.366706453 podStartE2EDuration="7.140040965s" totalImagesPullingTime="3.690447413s" totalInitContainerRuntime="1.082887099s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:48:41.138555344 +0000 UTC m=+357.648153243" watchObservedRunningTime="2026-07-19 07:48:41.140040965 +0000 UTC m=+357.649638865"
Jul 19 07:48:50.115160 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:50.115129    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-747d9ff956-clwgr"
Jul 19 07:48:52.125252 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:48:52.125220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-789467ff8c-mz9r9"
Jul 19 07:49:03.032873 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.032838    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"]
Jul 19 07:49:03.043132 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.043103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.046813 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.046379    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 19 07:49:03.046813 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.046650    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fxkxg\""
Jul 19 07:49:03.047336 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.047186    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 19 07:49:03.051662 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.049165    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 19 07:49:03.060278 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.060242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"]
Jul 19 07:49:03.094780 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.094925 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.094925 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095042 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095042 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095042 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.094997    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97b5cadb-546b-4c39-9863-cbc4ef983810-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095042 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.095036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ljp\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-kube-api-access-w6ljp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095187 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.095070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.095187 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.095145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196148 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196112    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196148 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196199    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97b5cadb-546b-4c39-9863-cbc4ef983810-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ljp\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-kube-api-access-w6ljp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196388 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.196618 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.196404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.197058 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.197032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.197223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.197068    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.197223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.197133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.197337 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.197313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.197523 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.197493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97b5cadb-546b-4c39-9863-cbc4ef983810-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.200387 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.200354    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.203507 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.203464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.205016 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.204988    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ljp\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-kube-api-access-w6ljp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.205283 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.205261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97b5cadb-546b-4c39-9863-cbc4ef983810-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8rr4\" (UID: \"97b5cadb-546b-4c39-9863-cbc4ef983810\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.360905 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.360877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:03.525626 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:03.525596    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"]
Jul 19 07:49:03.529345 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:49:03.529310    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b5cadb_546b_4c39_9863_cbc4ef983810.slice/crio-f2c1e9ac47f8b73687af906ba56f8a3eb1a010ec405c334096b45d3118dadf81 WatchSource:0}: Error finding container f2c1e9ac47f8b73687af906ba56f8a3eb1a010ec405c334096b45d3118dadf81: Status 404 returned error can't find the container with id f2c1e9ac47f8b73687af906ba56f8a3eb1a010ec405c334096b45d3118dadf81
Jul 19 07:49:04.228507 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.228466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4" event={"ID":"97b5cadb-546b-4c39-9863-cbc4ef983810","Type":"ContainerStarted","Data":"f2c1e9ac47f8b73687af906ba56f8a3eb1a010ec405c334096b45d3118dadf81"}
Jul 19 07:49:04.870207 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.870168    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"]
Jul 19 07:49:04.873648 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.873623    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:04.876473 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.876429    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lnp6j\""
Jul 19 07:49:04.876593 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.876492    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 19 07:49:04.876593 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.876492    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 19 07:49:04.876593 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.876492    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 19 07:49:04.887005 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:04.886981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"]
Jul 19 07:49:05.014112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014077    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014129    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8j7p\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-kube-api-access-h8j7p\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.014529 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.014305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.114963 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.114920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.114963 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.114963    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.115213 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.114981    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8j7p\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-kube-api-access-h8j7p\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.115213 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.115000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.115213 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.115056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.115213 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.115194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.115418 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.115247    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.116231 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.116200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.118359 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.118328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.118477 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.118331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.118477 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.118387    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.118477 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.118411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.124500 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.124448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.124500 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.124463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8j7p\" (UniqueName: \"kubernetes.io/projected/00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19-kube-api-access-h8j7p\") pod \"istiod-openshift-gateway-94bb8fbfd-5kph2\" (UID: \"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:05.185058 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:05.185026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:06.267463 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:06.267429    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:49:06.267769 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:06.267500    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:49:06.267769 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:06.267528    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:49:06.380842 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:06.380812    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"]
Jul 19 07:49:06.381865 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:49:06.381828    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fe4a5d_765b_4d27_a5d6_ef2ba62b0d19.slice/crio-879d22d627bac957adc276dfd942d87f3dcdf5bff7594209f09428aab884684f WatchSource:0}: Error finding container 879d22d627bac957adc276dfd942d87f3dcdf5bff7594209f09428aab884684f: Status 404 returned error can't find the container with id 879d22d627bac957adc276dfd942d87f3dcdf5bff7594209f09428aab884684f
Jul 19 07:49:07.249143 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:07.249097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4" event={"ID":"97b5cadb-546b-4c39-9863-cbc4ef983810","Type":"ContainerStarted","Data":"1212a763fbe581d6d21e0b309dbc51a4215c229f165f2e57cb0c8ddae66ea1a0"}
Jul 19 07:49:07.250568 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:07.250531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2" event={"ID":"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19","Type":"ContainerStarted","Data":"879d22d627bac957adc276dfd942d87f3dcdf5bff7594209f09428aab884684f"}
Jul 19 07:49:07.284886 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:07.284839    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4" podStartSLOduration=2.549318092 podStartE2EDuration="5.284825938s" totalImagesPullingTime="2.735507846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:07.283217402 +0000 UTC m=+383.792815304" watchObservedRunningTime="2026-07-19 07:49:07.284825938 +0000 UTC m=+383.794423836"
Jul 19 07:49:07.361211 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:07.361147    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:07.366832 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:07.366807    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:08.256437 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:08.256407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:08.257509 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:08.257477    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8rr4"
Jul 19 07:49:08.442958 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:08.442924    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:49:08.443244 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:08.442989    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:49:09.262213 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:09.262170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2" event={"ID":"00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19","Type":"ContainerStarted","Data":"a1016a1a2c7de986945f26d7e7ecff7da6ef1df9a7d7f9af1024bed7acdcbdfc"}
Jul 19 07:49:09.262376 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:09.262232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:09.263925 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:09.263897    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2"
Jul 19 07:49:09.303762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:49:09.303715    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kph2" podStartSLOduration=3.245116815 podStartE2EDuration="5.303661377s" totalImagesPullingTime="2.058544562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:49:09.301942588 +0000 UTC m=+385.811540487" watchObservedRunningTime="2026-07-19 07:49:09.303661377 +0000 UTC m=+385.813259277"
Jul 19 07:50:11.434080 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.434001    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"]
Jul 19 07:50:11.437316 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.437299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.443510 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.443480    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k4wmr\""
Jul 19 07:50:11.443725 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.443668    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 19 07:50:11.444786 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.444757    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 19 07:50:11.446948 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.446923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"]
Jul 19 07:50:11.561245 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.561217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vz68\" (UniqueName: \"kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.561430 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.561320    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.561430 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.561342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.662577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.662537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.662577 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.662583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.662831 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.662720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vz68\" (UniqueName: \"kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.662969 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.662953    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.663006 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.662978    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.670786 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.670762    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vz68\" (UniqueName: \"kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.752684 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.752610    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:11.879315 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:11.879270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"]
Jul 19 07:50:11.882145 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:50:11.882117    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf316e89e_10f7_4a63_8b22_de9b8180f2d9.slice/crio-405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e WatchSource:0}: Error finding container 405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e: Status 404 returned error can't find the container with id 405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e
Jul 19 07:50:12.027720 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.027634    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"]
Jul 19 07:50:12.030923 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.030907    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.039110 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.039085    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"]
Jul 19 07:50:12.165832 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.165796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.166003 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.165855    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cth7\" (UniqueName: \"kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.166003 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.165919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.266484 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.266445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.266484 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.266492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cth7\" (UniqueName: \"kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.266721 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.266516    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.266817 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.266800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.266877 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.266824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.274385 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.274367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cth7\" (UniqueName: \"kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.361694 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.361665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:12.437407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.437376    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"]
Jul 19 07:50:12.441025 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.441001    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.451380 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.451344    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"]
Jul 19 07:50:12.494455 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.494433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"]
Jul 19 07:50:12.496535 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:50:12.496512    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297dc5c4_78a4_4ff8_a940_0c99f99f847d.slice/crio-2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4 WatchSource:0}: Error finding container 2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4: Status 404 returned error can't find the container with id 2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4
Jul 19 07:50:12.557250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.557219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerStarted","Data":"00d63656554334bab2d5e8fc8e97e286daa2cdc0a479823387061d7ba3866a67"}
Jul 19 07:50:12.557385 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.557272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerStarted","Data":"2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4"}
Jul 19 07:50:12.558787 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.558760    2559 generic.go:356] "Generic (PLEG): container finished" podID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerID="dbc72aa2bd516de5d57de32d53b043c863a0772787a242faae74670b3e0a955b" exitCode=0
Jul 19 07:50:12.558890 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.558805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f" event={"ID":"f316e89e-10f7-4a63-8b22-de9b8180f2d9","Type":"ContainerDied","Data":"dbc72aa2bd516de5d57de32d53b043c863a0772787a242faae74670b3e0a955b"}
Jul 19 07:50:12.558890 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.558846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f" event={"ID":"f316e89e-10f7-4a63-8b22-de9b8180f2d9","Type":"ContainerStarted","Data":"405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e"}
Jul 19 07:50:12.569481 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.569457    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.569574 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.569518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.569616 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.569600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g2r\" (UniqueName: \"kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.670339 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.670262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.670472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.670378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g2r\" (UniqueName: \"kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.670472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.670418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.670681 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.670655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.670788 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.670744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.679579 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.679556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g2r\" (UniqueName: \"kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.753635 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.753605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:12.877784 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:12.877754    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"]
Jul 19 07:50:12.879711 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:50:12.879668    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20af166_298d_453d_8f22_f6f7048d8566.slice/crio-cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8 WatchSource:0}: Error finding container cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8: Status 404 returned error can't find the container with id cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8
Jul 19 07:50:13.032880 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.032837    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"]
Jul 19 07:50:13.036215 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.036197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.044027 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.044000    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"]
Jul 19 07:50:13.174639 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.174603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.174812 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.174727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.174812 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.174758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqjv\" (UniqueName: \"kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.275684 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.275595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.275684 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.275636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqjv\" (UniqueName: \"kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.275885 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.275686    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.276001 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.275981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.276091 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.276074    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.284223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.284201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqjv\" (UniqueName: \"kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.351979 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.351953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:13.566511 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.566267    2559 generic.go:356] "Generic (PLEG): container finished" podID="c20af166-298d-453d-8f22-f6f7048d8566" containerID="f6eccec77f72e6db1b65b2f500e3e063849c873b909186e89f88496657ff1704" exitCode=0
Jul 19 07:50:13.566511 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.566405    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw" event={"ID":"c20af166-298d-453d-8f22-f6f7048d8566","Type":"ContainerDied","Data":"f6eccec77f72e6db1b65b2f500e3e063849c873b909186e89f88496657ff1704"}
Jul 19 07:50:13.566511 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.566452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw" event={"ID":"c20af166-298d-453d-8f22-f6f7048d8566","Type":"ContainerStarted","Data":"cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8"}
Jul 19 07:50:13.568376 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.568272    2559 generic.go:356] "Generic (PLEG): container finished" podID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerID="00d63656554334bab2d5e8fc8e97e286daa2cdc0a479823387061d7ba3866a67" exitCode=0
Jul 19 07:50:13.568376 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.568328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerDied","Data":"00d63656554334bab2d5e8fc8e97e286daa2cdc0a479823387061d7ba3866a67"}
Jul 19 07:50:13.600161 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:13.600138    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"]
Jul 19 07:50:13.602274 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:50:13.602241    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8d8478_01f5_4341_9802_eb7138a2c7b3.slice/crio-820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091 WatchSource:0}: Error finding container 820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091: Status 404 returned error can't find the container with id 820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091
Jul 19 07:50:14.574409 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.574377    2559 generic.go:356] "Generic (PLEG): container finished" podID="c20af166-298d-453d-8f22-f6f7048d8566" containerID="46bb0a3eb550a48605165928d11ae1aa1a9194a40b68fe9f24ac03fd5bf6f6c6" exitCode=0
Jul 19 07:50:14.574759 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.574462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw" event={"ID":"c20af166-298d-453d-8f22-f6f7048d8566","Type":"ContainerDied","Data":"46bb0a3eb550a48605165928d11ae1aa1a9194a40b68fe9f24ac03fd5bf6f6c6"}
Jul 19 07:50:14.576412 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.576389    2559 generic.go:356] "Generic (PLEG): container finished" podID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerID="24d0f495a667b995d279d0e44c64b27ae0fd45b987426405d56ca7d303e5b645" exitCode=0
Jul 19 07:50:14.576479 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.576463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f" event={"ID":"f316e89e-10f7-4a63-8b22-de9b8180f2d9","Type":"ContainerDied","Data":"24d0f495a667b995d279d0e44c64b27ae0fd45b987426405d56ca7d303e5b645"}
Jul 19 07:50:14.578489 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.578470    2559 generic.go:356] "Generic (PLEG): container finished" podID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerID="8b7b9b7f46f95c98420a855b5e8609ae3e1e6c9c75e6c4cc9d899a78d0ecfbae" exitCode=0
Jul 19 07:50:14.578587 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.578531    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerDied","Data":"8b7b9b7f46f95c98420a855b5e8609ae3e1e6c9c75e6c4cc9d899a78d0ecfbae"}
Jul 19 07:50:14.580120 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.580100    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerID="e4f20845a99ec9545be6e7598b1415f922409da481de2c6b4a9e9bee9c5e7937" exitCode=0
Jul 19 07:50:14.580207 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.580147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p" event={"ID":"1d8d8478-01f5-4341-9802-eb7138a2c7b3","Type":"ContainerDied","Data":"e4f20845a99ec9545be6e7598b1415f922409da481de2c6b4a9e9bee9c5e7937"}
Jul 19 07:50:14.580207 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:14.580180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p" event={"ID":"1d8d8478-01f5-4341-9802-eb7138a2c7b3","Type":"ContainerStarted","Data":"820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091"}
Jul 19 07:50:15.588150 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.588117    2559 generic.go:356] "Generic (PLEG): container finished" podID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerID="832215e731b8d31141dc8bcda79d2ba3da007e6d44b743d837ea2033582d8d1b" exitCode=0
Jul 19 07:50:15.588563 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.588190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerDied","Data":"832215e731b8d31141dc8bcda79d2ba3da007e6d44b743d837ea2033582d8d1b"}
Jul 19 07:50:15.589869 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.589843    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerID="1706028e9b09e883cab47f3a9d4a97c42cc94c76952e5870dd8b0e604e99a7a0" exitCode=0
Jul 19 07:50:15.589998 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.589924    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p" event={"ID":"1d8d8478-01f5-4341-9802-eb7138a2c7b3","Type":"ContainerDied","Data":"1706028e9b09e883cab47f3a9d4a97c42cc94c76952e5870dd8b0e604e99a7a0"}
Jul 19 07:50:15.592268 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.592248    2559 generic.go:356] "Generic (PLEG): container finished" podID="c20af166-298d-453d-8f22-f6f7048d8566" containerID="b63208e9fec1d6e8d2f8b22050566aa662fcf0ce654e220cd6c1aa2669986a29" exitCode=0
Jul 19 07:50:15.592354 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.592329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw" event={"ID":"c20af166-298d-453d-8f22-f6f7048d8566","Type":"ContainerDied","Data":"b63208e9fec1d6e8d2f8b22050566aa662fcf0ce654e220cd6c1aa2669986a29"}
Jul 19 07:50:15.594321 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.594299    2559 generic.go:356] "Generic (PLEG): container finished" podID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerID="79c57da2b94469339f2cf417164be55027844ebadaaa4a18931270141a120fe9" exitCode=0
Jul 19 07:50:15.594412 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:15.594339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f" event={"ID":"f316e89e-10f7-4a63-8b22-de9b8180f2d9","Type":"ContainerDied","Data":"79c57da2b94469339f2cf417164be55027844ebadaaa4a18931270141a120fe9"}
Jul 19 07:50:16.600960 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.600923    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerID="97e2bb53071d09c6dcc291771f2c4a281525201bd4324eff59768c3915284305" exitCode=0
Jul 19 07:50:16.601467 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.601038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p" event={"ID":"1d8d8478-01f5-4341-9802-eb7138a2c7b3","Type":"ContainerDied","Data":"97e2bb53071d09c6dcc291771f2c4a281525201bd4324eff59768c3915284305"}
Jul 19 07:50:16.747312 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.747273    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:16.792302 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.792281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:16.795312 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.795289    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:16.908661 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908571    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util\") pod \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") "
Jul 19 07:50:16.908661 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908622    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8g2r\" (UniqueName: \"kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r\") pod \"c20af166-298d-453d-8f22-f6f7048d8566\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") "
Jul 19 07:50:16.908661 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908665    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util\") pod \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908738    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util\") pod \"c20af166-298d-453d-8f22-f6f7048d8566\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908769    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cth7\" (UniqueName: \"kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7\") pod \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908805    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle\") pod \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908824    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vz68\" (UniqueName: \"kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68\") pod \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\" (UID: \"f316e89e-10f7-4a63-8b22-de9b8180f2d9\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908859    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle\") pod \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\" (UID: \"297dc5c4-78a4-4ff8-a940-0c99f99f847d\") "
Jul 19 07:50:16.908967 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.908881    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle\") pod \"c20af166-298d-453d-8f22-f6f7048d8566\" (UID: \"c20af166-298d-453d-8f22-f6f7048d8566\") "
Jul 19 07:50:16.909501 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.909462    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle" (OuterVolumeSpecName: "bundle") pod "f316e89e-10f7-4a63-8b22-de9b8180f2d9" (UID: "f316e89e-10f7-4a63-8b22-de9b8180f2d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:16.909873 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.909844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle" (OuterVolumeSpecName: "bundle") pod "c20af166-298d-453d-8f22-f6f7048d8566" (UID: "c20af166-298d-453d-8f22-f6f7048d8566"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:16.910130 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.910105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle" (OuterVolumeSpecName: "bundle") pod "297dc5c4-78a4-4ff8-a940-0c99f99f847d" (UID: "297dc5c4-78a4-4ff8-a940-0c99f99f847d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:16.911590 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.911557    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r" (OuterVolumeSpecName: "kube-api-access-n8g2r") pod "c20af166-298d-453d-8f22-f6f7048d8566" (UID: "c20af166-298d-453d-8f22-f6f7048d8566"). InnerVolumeSpecName "kube-api-access-n8g2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:50:16.911679 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.911582    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68" (OuterVolumeSpecName: "kube-api-access-7vz68") pod "f316e89e-10f7-4a63-8b22-de9b8180f2d9" (UID: "f316e89e-10f7-4a63-8b22-de9b8180f2d9"). InnerVolumeSpecName "kube-api-access-7vz68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:50:16.911679 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.911599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7" (OuterVolumeSpecName: "kube-api-access-6cth7") pod "297dc5c4-78a4-4ff8-a940-0c99f99f847d" (UID: "297dc5c4-78a4-4ff8-a940-0c99f99f847d"). InnerVolumeSpecName "kube-api-access-6cth7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:50:16.915214 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.915191    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util" (OuterVolumeSpecName: "util") pod "f316e89e-10f7-4a63-8b22-de9b8180f2d9" (UID: "f316e89e-10f7-4a63-8b22-de9b8180f2d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:16.915640 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.915619    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util" (OuterVolumeSpecName: "util") pod "c20af166-298d-453d-8f22-f6f7048d8566" (UID: "c20af166-298d-453d-8f22-f6f7048d8566"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:16.915755 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:16.915736    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util" (OuterVolumeSpecName: "util") pod "297dc5c4-78a4-4ff8-a940-0c99f99f847d" (UID: "297dc5c4-78a4-4ff8-a940-0c99f99f847d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:17.010130 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010098    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8g2r\" (UniqueName: \"kubernetes.io/projected/c20af166-298d-453d-8f22-f6f7048d8566-kube-api-access-n8g2r\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010130 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010126    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010130 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010137    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010146    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cth7\" (UniqueName: \"kubernetes.io/projected/297dc5c4-78a4-4ff8-a940-0c99f99f847d-kube-api-access-6cth7\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010156    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f316e89e-10f7-4a63-8b22-de9b8180f2d9-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010164    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vz68\" (UniqueName: \"kubernetes.io/projected/f316e89e-10f7-4a63-8b22-de9b8180f2d9-kube-api-access-7vz68\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010172    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010179    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c20af166-298d-453d-8f22-f6f7048d8566-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.010329 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.010186    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/297dc5c4-78a4-4ff8-a940-0c99f99f847d-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.608576 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.608546    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc"
Jul 19 07:50:17.608576 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.608570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc" event={"ID":"297dc5c4-78a4-4ff8-a940-0c99f99f847d","Type":"ContainerDied","Data":"2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4"}
Jul 19 07:50:17.609061 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.608618    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2472e0204115e7939aeff6c2f61e5de5c21d22a7315fda972dbe8c742dfdfcb4"
Jul 19 07:50:17.610437 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.610416    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw"
Jul 19 07:50:17.610572 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.610436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw" event={"ID":"c20af166-298d-453d-8f22-f6f7048d8566","Type":"ContainerDied","Data":"cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8"}
Jul 19 07:50:17.610572 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.610469    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb18f3c1c842888047096788899801fe916c2e3ca8bfd5cd0d2a69f0933390d8"
Jul 19 07:50:17.612355 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.612331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f" event={"ID":"f316e89e-10f7-4a63-8b22-de9b8180f2d9","Type":"ContainerDied","Data":"405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e"}
Jul 19 07:50:17.612474 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.612372    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="405fd5fefbd01b790e5cdddf16b100e0080052161647cb89259572443f90dd9e"
Jul 19 07:50:17.612474 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.612457    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f"
Jul 19 07:50:17.730931 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.730907    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:17.816559 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.816533    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle\") pod \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") "
Jul 19 07:50:17.816762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.816573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util\") pod \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") "
Jul 19 07:50:17.816762 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.816621    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcqjv\" (UniqueName: \"kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv\") pod \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\" (UID: \"1d8d8478-01f5-4341-9802-eb7138a2c7b3\") "
Jul 19 07:50:17.817079 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.817056    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle" (OuterVolumeSpecName: "bundle") pod "1d8d8478-01f5-4341-9802-eb7138a2c7b3" (UID: "1d8d8478-01f5-4341-9802-eb7138a2c7b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:17.818764 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.818731    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv" (OuterVolumeSpecName: "kube-api-access-zcqjv") pod "1d8d8478-01f5-4341-9802-eb7138a2c7b3" (UID: "1d8d8478-01f5-4341-9802-eb7138a2c7b3"). InnerVolumeSpecName "kube-api-access-zcqjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:50:17.821773 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.821749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util" (OuterVolumeSpecName: "util") pod "1d8d8478-01f5-4341-9802-eb7138a2c7b3" (UID: "1d8d8478-01f5-4341-9802-eb7138a2c7b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 07:50:17.917233 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.917157    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcqjv\" (UniqueName: \"kubernetes.io/projected/1d8d8478-01f5-4341-9802-eb7138a2c7b3-kube-api-access-zcqjv\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.917233 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.917183    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:17.917233 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:17.917194    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d8d8478-01f5-4341-9802-eb7138a2c7b3-util\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:50:18.619938 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:18.619903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p" event={"ID":"1d8d8478-01f5-4341-9802-eb7138a2c7b3","Type":"ContainerDied","Data":"820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091"}
Jul 19 07:50:18.619938 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:18.619927    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p"
Jul 19 07:50:18.620349 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:18.619951    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="820c05fabb3a769626995515c1bb982d40c308e2087863feaa600bfdc61e9091"
Jul 19 07:50:24.359694 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.359661    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f5965b578-w7wxx"]
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360008    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360022    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360036    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360042    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360049    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360055    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360064    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="pull"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360070    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="pull"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360076    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360081    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360089    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360095    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="util"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360105    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360114    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="extract"
Jul 19 07:50:24.360112 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360120    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360125    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360132    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360137    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360145    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360149    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="pull"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360155    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="util"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360160    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="util"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360165    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="extract"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360170    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="extract"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360229    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="297dc5c4-78a4-4ff8-a940-0c99f99f847d" containerName="extract"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360239    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f316e89e-10f7-4a63-8b22-de9b8180f2d9" containerName="extract"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360245    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20af166-298d-453d-8f22-f6f7048d8566" containerName="extract"
Jul 19 07:50:24.360676 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.360250    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8d8478-01f5-4341-9802-eb7138a2c7b3" containerName="extract"
Jul 19 07:50:24.363374 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.363349    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.374142 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.374116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f5965b578-w7wxx"]
Jul 19 07:50:24.473566 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-oauth-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473566 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-service-ca\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473782 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473782 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473782 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-trusted-ca-bundle\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473782 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-oauth-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.473782 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.473782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ggj\" (UniqueName: \"kubernetes.io/projected/ec79ac7b-7cfc-4965-bcf6-b278837440ae-kube-api-access-h7ggj\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574512 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-trusted-ca-bundle\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574558    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-oauth-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574577    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ggj\" (UniqueName: \"kubernetes.io/projected/ec79ac7b-7cfc-4965-bcf6-b278837440ae-kube-api-access-h7ggj\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574680    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-oauth-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.574754 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-service-ca\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.575048 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.574791    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.575301 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.575276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.575546 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.575516    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-trusted-ca-bundle\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.575734 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.575684    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-oauth-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.575788 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.575730    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec79ac7b-7cfc-4965-bcf6-b278837440ae-service-ca\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.577245 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.577224    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-serving-cert\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.577394 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.577375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec79ac7b-7cfc-4965-bcf6-b278837440ae-console-oauth-config\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.584741 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.584692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ggj\" (UniqueName: \"kubernetes.io/projected/ec79ac7b-7cfc-4965-bcf6-b278837440ae-kube-api-access-h7ggj\") pod \"console-f5965b578-w7wxx\" (UID: \"ec79ac7b-7cfc-4965-bcf6-b278837440ae\") " pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.673064 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.672974    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:24.808365 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:24.808339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f5965b578-w7wxx"]
Jul 19 07:50:24.810756 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:50:24.810726    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec79ac7b_7cfc_4965_bcf6_b278837440ae.slice/crio-743f596572f8b152e82d4f00e736a43353de2622fd71285afb1a34ca46d49e46 WatchSource:0}: Error finding container 743f596572f8b152e82d4f00e736a43353de2622fd71285afb1a34ca46d49e46: Status 404 returned error can't find the container with id 743f596572f8b152e82d4f00e736a43353de2622fd71285afb1a34ca46d49e46
Jul 19 07:50:25.656029 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:25.655988    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5965b578-w7wxx" event={"ID":"ec79ac7b-7cfc-4965-bcf6-b278837440ae","Type":"ContainerStarted","Data":"2d34ce855f88beaa3661cb2c169a31b72bf4606254b0c885ff707d84e6d2e6b4"}
Jul 19 07:50:25.656415 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:25.656080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f5965b578-w7wxx" event={"ID":"ec79ac7b-7cfc-4965-bcf6-b278837440ae","Type":"ContainerStarted","Data":"743f596572f8b152e82d4f00e736a43353de2622fd71285afb1a34ca46d49e46"}
Jul 19 07:50:25.677048 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:25.676996    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f5965b578-w7wxx" podStartSLOduration=1.676981362 podStartE2EDuration="1.676981362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:50:25.675102253 +0000 UTC m=+462.184700153" watchObservedRunningTime="2026-07-19 07:50:25.676981362 +0000 UTC m=+462.186579261"
Jul 19 07:50:34.673489 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:34.673452    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:34.673941 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:34.673512    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:34.679269 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:34.679248    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:34.709616 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:34.709593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f5965b578-w7wxx"
Jul 19 07:50:34.768243 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:34.768208    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:50:59.789523 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:50:59.789476    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-669d468d7c-296mw" podUID="43df65af-932f-4565-9009-86a5af68256a" containerName="console" containerID="cri-o://fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2" gracePeriod=15
Jul 19 07:51:00.030141 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.030118    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:51:00.198739 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198690    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.198936 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.198936 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.198936 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.198936 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198923    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.199148 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198947    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.199148 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.198987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6t28\" (UniqueName: \"kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28\") pod \"43df65af-932f-4565-9009-86a5af68256a\" (UID: \"43df65af-932f-4565-9009-86a5af68256a\") "
Jul 19 07:51:00.199369 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.199335    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:51:00.199369 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.199344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca" (OuterVolumeSpecName: "service-ca") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:51:00.199526 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.199384    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config" (OuterVolumeSpecName: "console-config") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:51:00.199526 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.199448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 07:51:00.201179 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.201155    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28" (OuterVolumeSpecName: "kube-api-access-d6t28") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "kube-api-access-d6t28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 07:51:00.201512 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.201496    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:51:00.201559 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.201523    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43df65af-932f-4565-9009-86a5af68256a" (UID: "43df65af-932f-4565-9009-86a5af68256a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 19 07:51:00.300044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300005    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300036    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-console-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300044 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300049    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300272 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300061    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-oauth-serving-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300272 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300074    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6t28\" (UniqueName: \"kubernetes.io/projected/43df65af-932f-4565-9009-86a5af68256a-kube-api-access-d6t28\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300272 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300086    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43df65af-932f-4565-9009-86a5af68256a-trusted-ca-bundle\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.300272 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.300096    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43df65af-932f-4565-9009-86a5af68256a-console-oauth-config\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 07:51:00.832620 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.832582    2559 generic.go:356] "Generic (PLEG): container finished" podID="43df65af-932f-4565-9009-86a5af68256a" containerID="fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2" exitCode=0
Jul 19 07:51:00.833103 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.832641    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-669d468d7c-296mw"
Jul 19 07:51:00.833103 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.832671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d468d7c-296mw" event={"ID":"43df65af-932f-4565-9009-86a5af68256a","Type":"ContainerDied","Data":"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"}
Jul 19 07:51:00.833103 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.832785    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-669d468d7c-296mw" event={"ID":"43df65af-932f-4565-9009-86a5af68256a","Type":"ContainerDied","Data":"cb0af952edcf4fd30f0360087b5b6f7c58ef77b83f640b1c3d9a55c772aab635"}
Jul 19 07:51:00.833103 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.832815    2559 scope.go:117] "RemoveContainer" containerID="fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"
Jul 19 07:51:00.841007 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.840985    2559 scope.go:117] "RemoveContainer" containerID="fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"
Jul 19 07:51:00.841280 ip-10-0-129-91 kubenswrapper[2559]: E0719 07:51:00.841261    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2\": container with ID starting with fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2 not found: ID does not exist" containerID="fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"
Jul 19 07:51:00.841331 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.841288    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2"} err="failed to get container status \"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2\": rpc error: code = NotFound desc = could not find container \"fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2\": container with ID starting with fd0d918950c5e7f393b9ae95076b812a24b9aa1e8aba85e270322470ffce20e2 not found: ID does not exist"
Jul 19 07:51:00.859864 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.859837    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:51:00.861680 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:00.861658    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-669d468d7c-296mw"]
Jul 19 07:51:02.460713 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:02.460670    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43df65af-932f-4565-9009-86a5af68256a" path="/var/lib/kubelet/pods/43df65af-932f-4565-9009-86a5af68256a/volumes"
Jul 19 07:51:21.530141 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.530102    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"]
Jul 19 07:51:21.530606 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.530587    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43df65af-932f-4565-9009-86a5af68256a" containerName="console"
Jul 19 07:51:21.530673 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.530608    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43df65af-932f-4565-9009-86a5af68256a" containerName="console"
Jul 19 07:51:21.530756 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.530740    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43df65af-932f-4565-9009-86a5af68256a" containerName="console"
Jul 19 07:51:21.537513 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.536026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.538906 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.538886    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tkn4z\""
Jul 19 07:51:21.543631 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.543602    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"]
Jul 19 07:51:21.582845 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.582806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583012 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.582856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628628a5-603d-4637-b044-99b658969e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583012 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.582875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583086 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583086 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583156 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7skt\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-kube-api-access-m7skt\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583156 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583156 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.583246 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.583159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628628a5-603d-4637-b044-99b658969e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.683868 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.683827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684032 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.683881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7skt\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-kube-api-access-m7skt\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684032 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628628a5-603d-4637-b044-99b658969e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684253 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684314 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628628a5-603d-4637-b044-99b658969e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684368 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684368 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684397    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684432    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684737 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.684930 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.684907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628628a5-603d-4637-b044-99b658969e7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.686494 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.686469    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628628a5-603d-4637-b044-99b658969e7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.686670 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.686654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628628a5-603d-4637-b044-99b658969e7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.691182 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.691155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.691279 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.691264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7skt\" (UniqueName: \"kubernetes.io/projected/628628a5-603d-4637-b044-99b658969e7c-kube-api-access-m7skt\") pod \"maas-default-gateway-openshift-default-5896d4dd49-9j6kr\" (UID: \"628628a5-603d-4637-b044-99b658969e7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.849636 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.849605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:21.981254 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.981230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"]
Jul 19 07:51:21.983651 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:51:21.983625    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628628a5_603d_4637_b044_99b658969e7c.slice/crio-2fadf567bc6d342c54b84ffe5337975c01f73694f859cdc5deb6c656ede510bb WatchSource:0}: Error finding container 2fadf567bc6d342c54b84ffe5337975c01f73694f859cdc5deb6c656ede510bb: Status 404 returned error can't find the container with id 2fadf567bc6d342c54b84ffe5337975c01f73694f859cdc5deb6c656ede510bb
Jul 19 07:51:21.985734 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.985687    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:51:21.985839 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.985764    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:51:21.985839 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:21.985800    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:51:22.939180 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:22.939144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" event={"ID":"628628a5-603d-4637-b044-99b658969e7c","Type":"ContainerStarted","Data":"c640205f49dd60cacd31c3c5815f031a6e434196957a5d5133500262062765b2"}
Jul 19 07:51:22.939590 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:22.939193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" event={"ID":"628628a5-603d-4637-b044-99b658969e7c","Type":"ContainerStarted","Data":"2fadf567bc6d342c54b84ffe5337975c01f73694f859cdc5deb6c656ede510bb"}
Jul 19 07:51:22.959571 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:22.959522    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" podStartSLOduration=1.959505855 podStartE2EDuration="1.959505855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:51:22.95679365 +0000 UTC m=+519.466391550" watchObservedRunningTime="2026-07-19 07:51:22.959505855 +0000 UTC m=+519.469103754"
Jul 19 07:51:23.849750 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:23.849719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:23.854358 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:23.854330    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:23.945190 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:23.945160    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:51:23.946116 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:51:23.946094    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:52:55.820403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.820369    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"]
Jul 19 07:52:55.823423 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.823399    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:55.825947 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.825925    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-827ps\""
Jul 19 07:52:55.826050 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.825925    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 19 07:52:55.835199 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.835178    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"]
Jul 19 07:52:55.952669 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.952624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89r6\" (UniqueName: \"kubernetes.io/projected/be1159f9-c573-4ef0-91b1-341a118ec325-kube-api-access-k89r6\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:55.952863 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:55.952689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1159f9-c573-4ef0-91b1-341a118ec325-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.054119 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.054087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89r6\" (UniqueName: \"kubernetes.io/projected/be1159f9-c573-4ef0-91b1-341a118ec325-kube-api-access-k89r6\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.054280 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.054134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1159f9-c573-4ef0-91b1-341a118ec325-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.054802 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.054783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1159f9-c573-4ef0-91b1-341a118ec325-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.062061 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.062035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89r6\" (UniqueName: \"kubernetes.io/projected/be1159f9-c573-4ef0-91b1-341a118ec325-kube-api-access-k89r6\") pod \"payload-pre-processing-7677b774bb-9h6dv\" (UID: \"be1159f9-c573-4ef0-91b1-341a118ec325\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.133971 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.133896    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:56.269074 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.269051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"]
Jul 19 07:52:56.271148 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:52:56.271122    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1159f9_c573_4ef0_91b1_341a118ec325.slice/crio-6503425ebc0102d91db98b163e0ec54b1fdf61f77a02cf12fb48556a7f741d43 WatchSource:0}: Error finding container 6503425ebc0102d91db98b163e0ec54b1fdf61f77a02cf12fb48556a7f741d43: Status 404 returned error can't find the container with id 6503425ebc0102d91db98b163e0ec54b1fdf61f77a02cf12fb48556a7f741d43
Jul 19 07:52:56.393288 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:56.393199    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv" event={"ID":"be1159f9-c573-4ef0-91b1-341a118ec325","Type":"ContainerStarted","Data":"6503425ebc0102d91db98b163e0ec54b1fdf61f77a02cf12fb48556a7f741d43"}
Jul 19 07:52:58.406673 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:58.406581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv" event={"ID":"be1159f9-c573-4ef0-91b1-341a118ec325","Type":"ContainerStarted","Data":"9ba6e8e8b0c1870932a6f315ddf63aad46d7a6b379ec56efe34b333aff101445"}
Jul 19 07:52:58.406673 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:58.406666    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:52:58.427581 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:52:58.426163    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv" podStartSLOduration=1.5622126010000001 podStartE2EDuration="3.426144461s" totalImagesPullingTime="1.86393186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:52:58.424536431 +0000 UTC m=+614.934134332" watchObservedRunningTime="2026-07-19 07:52:58.426144461 +0000 UTC m=+614.935742361"
Jul 19 07:53:03.433133 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.433062    2559 generic.go:356] "Generic (PLEG): container finished" podID="628628a5-603d-4637-b044-99b658969e7c" containerID="c640205f49dd60cacd31c3c5815f031a6e434196957a5d5133500262062765b2" exitCode=0
Jul 19 07:53:03.433484 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.433142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" event={"ID":"628628a5-603d-4637-b044-99b658969e7c","Type":"ContainerDied","Data":"c640205f49dd60cacd31c3c5815f031a6e434196957a5d5133500262062765b2"}
Jul 19 07:53:03.433524 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.433500    2559 scope.go:117] "RemoveContainer" containerID="c640205f49dd60cacd31c3c5815f031a6e434196957a5d5133500262062765b2"
Jul 19 07:53:03.434497 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.434474    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:53:03.434560 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.434537    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:53:03.434594 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:03.434566    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 07:53:04.439495 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:04.439462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" event={"ID":"628628a5-603d-4637-b044-99b658969e7c","Type":"ContainerStarted","Data":"ef13a038bbc3bb478f109771338cd33effb52e02928c1845de3af67f8deca953"}
Jul 19 07:53:04.850285 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:04.850243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:53:04.851433 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:04.851407    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-9j6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 19 07:53:04.851553 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:04.851468    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" podUID="628628a5-603d-4637-b044-99b658969e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 19 07:53:05.850810 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:05.850728    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-9j6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 19 07:53:05.850810 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:05.850791    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" podUID="628628a5-603d-4637-b044-99b658969e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 19 07:53:06.850295 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:06.850255    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-9j6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 19 07:53:06.850472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:06.850312    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" podUID="628628a5-603d-4637-b044-99b658969e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 19 07:53:07.850529 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:07.850494    2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-9j6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 19 07:53:07.850926 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:07.850558    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr" podUID="628628a5-603d-4637-b044-99b658969e7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 19 07:53:08.854278 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:08.854249    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:53:08.854741 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:08.854540    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:53:08.855198 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:08.855181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-9j6kr"
Jul 19 07:53:09.413756 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:53:09.413728    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-9h6dv"
Jul 19 07:54:03.879288 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.879251    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"]
Jul 19 07:54:03.882751 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.882730    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.884995 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.884971    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 19 07:54:03.886031 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.886008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 19 07:54:03.886031 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.886021    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 19 07:54:03.886178 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.886114    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wc5bc\""
Jul 19 07:54:03.891656 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.891628    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"]
Jul 19 07:54:03.968498 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.968671 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.968671 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968577    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdvh\" (UniqueName: \"kubernetes.io/projected/6964e5d9-9ca6-4632-aac1-3b01465f632e-kube-api-access-ptdvh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.968671 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.968825 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:03.968825 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:03.968720    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6964e5d9-9ca6-4632-aac1-3b01465f632e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.069752 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.069918 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6964e5d9-9ca6-4632-aac1-3b01465f632e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.069918 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069811    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.069918 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.070076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdvh\" (UniqueName: \"kubernetes.io/projected/6964e5d9-9ca6-4632-aac1-3b01465f632e-kube-api-access-ptdvh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.070076 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.069950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.070169 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.070154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.070223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.070193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.070223 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.070214    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.072138 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.072111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6964e5d9-9ca6-4632-aac1-3b01465f632e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.072338 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.072323    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6964e5d9-9ca6-4632-aac1-3b01465f632e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.076756 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.076731    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdvh\" (UniqueName: \"kubernetes.io/projected/6964e5d9-9ca6-4632-aac1-3b01465f632e-kube-api-access-ptdvh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wprbj\" (UID: \"6964e5d9-9ca6-4632-aac1-3b01465f632e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.193309 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.193244    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:04.326273 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.326244    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"]
Jul 19 07:54:04.328487 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:54:04.328464    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6964e5d9_9ca6_4632_aac1_3b01465f632e.slice/crio-b51deb9acac302ad9b9f0fff87d3969c79805f1d1bc97af9661097190e57c4e7 WatchSource:0}: Error finding container b51deb9acac302ad9b9f0fff87d3969c79805f1d1bc97af9661097190e57c4e7: Status 404 returned error can't find the container with id b51deb9acac302ad9b9f0fff87d3969c79805f1d1bc97af9661097190e57c4e7
Jul 19 07:54:04.330256 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.330240    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 07:54:04.749087 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:04.749039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj" event={"ID":"6964e5d9-9ca6-4632-aac1-3b01465f632e","Type":"ContainerStarted","Data":"b51deb9acac302ad9b9f0fff87d3969c79805f1d1bc97af9661097190e57c4e7"}
Jul 19 07:54:05.756104 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:05.756066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj" event={"ID":"6964e5d9-9ca6-4632-aac1-3b01465f632e","Type":"ContainerStarted","Data":"363c97002131502cdd0cd076ca5a29d17eead8290bffd550b36173e9427428a2"}
Jul 19 07:54:05.774653 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:05.774605    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj" podStartSLOduration=1.801992309 podStartE2EDuration="2.77459062s" totalImagesPullingTime="972.598311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:54:05.771902576 +0000 UTC m=+682.281500474" watchObservedRunningTime="2026-07-19 07:54:05.77459062 +0000 UTC m=+682.284188533"
Jul 19 07:54:14.193374 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:14.193337    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:14.193773 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:14.193404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:14.205628 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:14.205605    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:14.819411 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:14.819376    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wprbj"
Jul 19 07:54:26.176788 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.176748    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"]
Jul 19 07:54:26.179978 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.179958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.182406 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.182387    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 19 07:54:26.191492 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.191467    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"]
Jul 19 07:54:26.268771 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268734    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.268940 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.268940 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684b8471-aa95-4f16-b635-bdfdb84bf809-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.268940 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268928    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.269096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268958    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.269096 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.268986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgtk\" (UniqueName: \"kubernetes.io/projected/684b8471-aa95-4f16-b635-bdfdb84bf809-kube-api-access-5sgtk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370135 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370094    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684b8471-aa95-4f16-b635-bdfdb84bf809-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370238    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370311 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgtk\" (UniqueName: \"kubernetes.io/projected/684b8471-aa95-4f16-b635-bdfdb84bf809-kube-api-access-5sgtk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370621 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370687 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370687 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.370815 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.370728    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.372828 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.372801    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/684b8471-aa95-4f16-b635-bdfdb84bf809-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.373020 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.373000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/684b8471-aa95-4f16-b635-bdfdb84bf809-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.377769 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.377747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgtk\" (UniqueName: \"kubernetes.io/projected/684b8471-aa95-4f16-b635-bdfdb84bf809-kube-api-access-5sgtk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz\" (UID: \"684b8471-aa95-4f16-b635-bdfdb84bf809\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.490449 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.490374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:26.615193 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.615162    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"]
Jul 19 07:54:26.616964 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:54:26.616937    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684b8471_aa95_4f16_b635_bdfdb84bf809.slice/crio-a9431fe62395aa32800ed4f6d3734b6113d155e11f7d2c06366ec93e216dae9e WatchSource:0}: Error finding container a9431fe62395aa32800ed4f6d3734b6113d155e11f7d2c06366ec93e216dae9e: Status 404 returned error can't find the container with id a9431fe62395aa32800ed4f6d3734b6113d155e11f7d2c06366ec93e216dae9e
Jul 19 07:54:26.865403 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.865366    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz" event={"ID":"684b8471-aa95-4f16-b635-bdfdb84bf809","Type":"ContainerStarted","Data":"4d19578574334e51ea6f60da867aefaf57db2ac8953800bffc3e2ed261866a5b"}
Jul 19 07:54:26.865539 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.865426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz" event={"ID":"684b8471-aa95-4f16-b635-bdfdb84bf809","Type":"ContainerStarted","Data":"a9431fe62395aa32800ed4f6d3734b6113d155e11f7d2c06366ec93e216dae9e"}
Jul 19 07:54:26.886552 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:26.886493    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz" podStartSLOduration=0.72625557 podStartE2EDuration="886.436055ms" totalImagesPullingTime="160.180485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:54:26.883528599 +0000 UTC m=+703.393126499" watchObservedRunningTime="2026-07-19 07:54:26.886436055 +0000 UTC m=+703.396033956"
Jul 19 07:54:34.470464 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.470383    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"]
Jul 19 07:54:34.474253 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.474229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.476645 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.476626    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 19 07:54:34.484896 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.484872    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"]
Jul 19 07:54:34.545131 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.545284 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntz6h\" (UniqueName: \"kubernetes.io/projected/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-kube-api-access-ntz6h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.545284 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545176    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.545407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.545407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.545407 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.545368    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.646963 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.646926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.646976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntz6h\" (UniqueName: \"kubernetes.io/projected/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-kube-api-access-ntz6h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647250 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.647883 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647835    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.648012 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.647946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.648226 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.648203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.649690 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.649670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.649901 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.649883    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.654295 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.654273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntz6h\" (UniqueName: \"kubernetes.io/projected/2fcaa8f2-5c86-435c-91b1-50b95cd4481c-kube-api-access-ntz6h\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r4vmq\" (UID: \"2fcaa8f2-5c86-435c-91b1-50b95cd4481c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.786535 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.786443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:34.917501 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:34.917479    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"]
Jul 19 07:54:34.919052 ip-10-0-129-91 kubenswrapper[2559]: W0719 07:54:34.919026    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fcaa8f2_5c86_435c_91b1_50b95cd4481c.slice/crio-96d247fe728f1794eb05cc1c620ec97fc0c424fc3b72fb5b93a4ed1aaddca320 WatchSource:0}: Error finding container 96d247fe728f1794eb05cc1c620ec97fc0c424fc3b72fb5b93a4ed1aaddca320: Status 404 returned error can't find the container with id 96d247fe728f1794eb05cc1c620ec97fc0c424fc3b72fb5b93a4ed1aaddca320
Jul 19 07:54:35.915515 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:35.915477    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq" event={"ID":"2fcaa8f2-5c86-435c-91b1-50b95cd4481c","Type":"ContainerStarted","Data":"ea8693c4b3b2646c133d2ec6254abb9db0dab5a5857267598573565a3be1624f"}
Jul 19 07:54:35.915914 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:35.915567    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq" event={"ID":"2fcaa8f2-5c86-435c-91b1-50b95cd4481c","Type":"ContainerStarted","Data":"96d247fe728f1794eb05cc1c620ec97fc0c424fc3b72fb5b93a4ed1aaddca320"}
Jul 19 07:54:35.932095 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:35.932047    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq" podStartSLOduration=1.7704217 podStartE2EDuration="1.932033617s" totalImagesPullingTime="161.611917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 07:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 07:54:35.93095822 +0000 UTC m=+712.440556120" watchObservedRunningTime="2026-07-19 07:54:35.932033617 +0000 UTC m=+712.441631515"
Jul 19 07:54:36.491472 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:36.491442    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:36.491638 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:36.491493    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:36.504278 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:36.504249    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:36.932332 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:36.932299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz"
Jul 19 07:54:44.786752 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:44.786720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:44.787126 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:44.786778    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:44.799013 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:44.798986    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 07:54:44.979192 ip-10-0-129-91 kubenswrapper[2559]: I0719 07:54:44.979164    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r4vmq"
Jul 19 08:10:20.365845 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.365813    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:10:20.369586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.369565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.372071 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.372050    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ef124b28-openshift-default-dockercfg-76z4l\""
Jul 19 08:10:20.388190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.388163    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:10:20.496368 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496332    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftd59\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496376    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496426    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496540    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496612    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.496775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.496633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597809 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftd59\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.597997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.597979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598344 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598344 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598338    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598612 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.598865 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.598844    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.600236 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.600218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.600390 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.600371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.618857 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.618794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftd59\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.624350 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.624326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token\") pod \"e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") " pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.681378 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.681347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:20.818721 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.818685    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:10:20.820352 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:10:20.820325    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96e3d4f_d2ad_4d9f_a544_ef331af0e711.slice/crio-bda02256b7a52792b542bc47b6b6a02eb23e5c262de38bd0ed8b2c151246f15d WatchSource:0}: Error finding container bda02256b7a52792b542bc47b6b6a02eb23e5c262de38bd0ed8b2c151246f15d: Status 404 returned error can't find the container with id bda02256b7a52792b542bc47b6b6a02eb23e5c262de38bd0ed8b2c151246f15d
Jul 19 08:10:20.822318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.822287    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:20.822402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.822347    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:20.822402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:20.822394    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:21.773199 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:21.773160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerStarted","Data":"c17a9938d974a8d59ef253603c76d839d3af487366abd63c922a3c8f4e126a06"}
Jul 19 08:10:21.773611 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:21.773224    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerStarted","Data":"bda02256b7a52792b542bc47b6b6a02eb23e5c262de38bd0ed8b2c151246f15d"}
Jul 19 08:10:21.797991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:21.797938    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podStartSLOduration=1.7979223100000001 podStartE2EDuration="1.79792231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:10:21.796599368 +0000 UTC m=+1658.306197267" watchObservedRunningTime="2026-07-19 08:10:21.79792231 +0000 UTC m=+1658.307520210"
Jul 19 08:10:22.681875 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:22.681838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:22.686666 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:22.686644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:22.781872 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:22.781846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:22.782886 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:22.782866    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:32.844631 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.844554    2559 generic.go:356] "Generic (PLEG): container finished" podID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerID="c17a9938d974a8d59ef253603c76d839d3af487366abd63c922a3c8f4e126a06" exitCode=0
Jul 19 08:10:32.844631 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.844591    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerDied","Data":"c17a9938d974a8d59ef253603c76d839d3af487366abd63c922a3c8f4e126a06"}
Jul 19 08:10:32.845052 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.845019    2559 scope.go:117] "RemoveContainer" containerID="c17a9938d974a8d59ef253603c76d839d3af487366abd63c922a3c8f4e126a06"
Jul 19 08:10:32.846072 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.846041    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:32.846127 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.846114    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:32.846162 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:32.846151    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 19 08:10:33.851600 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:33.851566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerStarted","Data":"2a0043bd8e38c8eb342b2af841cf1ee9d43664126365b46597c25ec3744852d8"}
Jul 19 08:10:34.682402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:34.682362    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:34.683506 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:34.683478    2559 patch_prober.go:28] interesting pod/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 19 08:10:34.683638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:34.683538    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 19 08:10:35.682772 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:35.682729    2559 patch_prober.go:28] interesting pod/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 19 08:10:35.683172 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:35.682825    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 19 08:10:36.682544 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:36.682505    2559 patch_prober.go:28] interesting pod/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 19 08:10:36.682745 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:36.682568    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 19 08:10:37.682451 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:37.682413    2559 patch_prober.go:28] interesting pod/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Jul 19 08:10:37.682901 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:37.682471    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Jul 19 08:10:37.782219 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:37.782186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:38.685780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:38.685747    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:10:38.686666 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:10:38.686643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:13:26.642017 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:26.641976    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:13:26.642563 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:26.642169    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy" containerID="cri-o://2a0043bd8e38c8eb342b2af841cf1ee9d43664126365b46597c25ec3744852d8" gracePeriod=30
Jul 19 08:13:31.788695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.788633    2559 generic.go:356] "Generic (PLEG): container finished" podID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerID="2a0043bd8e38c8eb342b2af841cf1ee9d43664126365b46597c25ec3744852d8" exitCode=0
Jul 19 08:13:31.789169 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.788671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerDied","Data":"2a0043bd8e38c8eb342b2af841cf1ee9d43664126365b46597c25ec3744852d8"}
Jul 19 08:13:31.789169 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.788830    2559 scope.go:117] "RemoveContainer" containerID="c17a9938d974a8d59ef253603c76d839d3af487366abd63c922a3c8f4e126a06"
Jul 19 08:13:31.903636 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.903576    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:13:31.949904 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.949874    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.949969    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950002    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950044    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftd59\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950073    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950099    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950125    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950412 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 08:13:31.950532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 08:13:31.950532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950473    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert\") pod \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\" (UID: \"f96e3d4f-d2ad-4d9f-a544-ef331af0e711\") "
Jul 19 08:13:31.950743 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950642    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data" (OuterVolumeSpecName: "istio-data") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 08:13:31.950815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950751    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 08:13:31.950933 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950911    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:13:31.951023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950960    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-data\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:31.951023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950981    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-credential-socket\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:31.951023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.950997    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-certs\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:31.951023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.951010    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-workload-socket\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:31.952418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.952393    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 19 08:13:31.952533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.952449    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token" (OuterVolumeSpecName: "istio-token") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 08:13:31.952788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.952763    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 19 08:13:31.952892 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:31.952839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59" (OuterVolumeSpecName: "kube-api-access-ftd59") pod "f96e3d4f-d2ad-4d9f-a544-ef331af0e711" (UID: "f96e3d4f-d2ad-4d9f-a544-ef331af0e711"). InnerVolumeSpecName "kube-api-access-ftd59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 19 08:13:32.052104 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.052067    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-token\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:32.052104 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.052097    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-podinfo\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:32.052104 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.052106    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftd59\" (UniqueName: \"kubernetes.io/projected/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-kube-api-access-ftd59\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:32.052332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.052117    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istio-envoy\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:32.052332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.052125    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96e3d4f-d2ad-4d9f-a544-ef331af0e711-istiod-ca-cert\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:13:32.795457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.795379    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"
Jul 19 08:13:32.795457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.795391    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k" event={"ID":"f96e3d4f-d2ad-4d9f-a544-ef331af0e711","Type":"ContainerDied","Data":"bda02256b7a52792b542bc47b6b6a02eb23e5c262de38bd0ed8b2c151246f15d"}
Jul 19 08:13:32.795457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.795443    2559 scope.go:117] "RemoveContainer" containerID="2a0043bd8e38c8eb342b2af841cf1ee9d43664126365b46597c25ec3744852d8"
Jul 19 08:13:32.813408 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.813379    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:13:32.816586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:32.816555    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ef124b28-openshift-default-5d648d46c5-9n87k"]
Jul 19 08:13:34.461664 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:34.461631    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" path="/var/lib/kubelet/pods/f96e3d4f-d2ad-4d9f-a544-ef331af0e711/volumes"
Jul 19 08:13:56.707282 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707245    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:13:56.707688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.707688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707600    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.707688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707610    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.707688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707615    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.707688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707685    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.707898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.707694    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f96e3d4f-d2ad-4d9f-a544-ef331af0e711" containerName="istio-proxy"
Jul 19 08:13:56.710743 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.710727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:13:56.713364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.713341    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:13:56.714435 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.714412    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:13:56.717590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.717571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:13:56.874800 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.874763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9\" (UID: \"2d7c278a-a8ff-4a4f-839f-96cbf0d55759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:13:56.975415 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.975332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9\" (UID: \"2d7c278a-a8ff-4a4f-839f-96cbf0d55759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:13:56.976004 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:56.975984    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9\" (UID: \"2d7c278a-a8ff-4a4f-839f-96cbf0d55759\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:13:57.022216 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:57.022193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:13:57.358049 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:57.358027    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:13:57.360088 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:13:57.360060    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7c278a_a8ff_4a4f_839f_96cbf0d55759.slice/crio-9383f41af4ecc022a59c03c87239e85954ba3f715adc6555a0b9e6249f2576e5 WatchSource:0}: Error finding container 9383f41af4ecc022a59c03c87239e85954ba3f715adc6555a0b9e6249f2576e5: Status 404 returned error can't find the container with id 9383f41af4ecc022a59c03c87239e85954ba3f715adc6555a0b9e6249f2576e5
Jul 19 08:13:57.361859 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:57.361841    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 19 08:13:57.926647 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:57.926611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerStarted","Data":"9383f41af4ecc022a59c03c87239e85954ba3f715adc6555a0b9e6249f2576e5"}
Jul 19 08:13:59.941157 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:59.941114    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerStarted","Data":"d8eb3733234c1b9c7ff14dab8daec395574a7f1592132a6cba82ea94f23c844b"}
Jul 19 08:13:59.955098 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:13:59.955045    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" podStartSLOduration=1.827601376 podStartE2EDuration="3.955027133s" totalImagesPullingTime="2.127425757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:13:59.954891738 +0000 UTC m=+1876.464489637" watchObservedRunningTime="2026-07-19 08:13:59.955027133 +0000 UTC m=+1876.464625033"
Jul 19 08:14:00.947213 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:00.947179    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerID="d8eb3733234c1b9c7ff14dab8daec395574a7f1592132a6cba82ea94f23c844b" exitCode=6
Jul 19 08:14:00.947609 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:00.947264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerDied","Data":"d8eb3733234c1b9c7ff14dab8daec395574a7f1592132a6cba82ea94f23c844b"}
Jul 19 08:14:00.947609 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:00.947469    2559 scope.go:117] "RemoveContainer" containerID="d8eb3733234c1b9c7ff14dab8daec395574a7f1592132a6cba82ea94f23c844b"
Jul 19 08:14:01.954734 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:01.954621    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerID="2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a" exitCode=6
Jul 19 08:14:01.954734 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:01.954716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerDied","Data":"2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a"}
Jul 19 08:14:01.955188 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:01.954775    2559 scope.go:117] "RemoveContainer" containerID="d8eb3733234c1b9c7ff14dab8daec395574a7f1592132a6cba82ea94f23c844b"
Jul 19 08:14:01.955188 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:01.954972    2559 scope.go:117] "RemoveContainer" containerID="2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a"
Jul 19 08:14:01.955255 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:01.955195    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9_odh-ai-gateway-infra(2d7c278a-a8ff-4a4f-839f-96cbf0d55759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759"
Jul 19 08:14:02.961275 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:02.961250    2559 scope.go:117] "RemoveContainer" containerID="2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a"
Jul 19 08:14:02.961641 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:02.961438    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9_odh-ai-gateway-infra(2d7c278a-a8ff-4a4f-839f-96cbf0d55759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759"
Jul 19 08:14:14.457474 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:14.457394    2559 scope.go:117] "RemoveContainer" containerID="2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a"
Jul 19 08:14:15.026903 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:15.026818    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerStarted","Data":"76f4b8b8b5a4e53661fd0d5a7102b80dead96b7ecdf368fcd64b749bdf30e79b"}
Jul 19 08:14:16.033553 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:16.033518    2559 generic.go:356] "Generic (PLEG): container finished" podID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerID="76f4b8b8b5a4e53661fd0d5a7102b80dead96b7ecdf368fcd64b749bdf30e79b" exitCode=6
Jul 19 08:14:16.034030 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:16.033597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerDied","Data":"76f4b8b8b5a4e53661fd0d5a7102b80dead96b7ecdf368fcd64b749bdf30e79b"}
Jul 19 08:14:16.034030 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:16.033677    2559 scope.go:117] "RemoveContainer" containerID="2fe5ef80e7a2d714668067462a7032399036e4ffb59b90f47aff28eceefe9d0a"
Jul 19 08:14:16.034030 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:16.033908    2559 scope.go:117] "RemoveContainer" containerID="76f4b8b8b5a4e53661fd0d5a7102b80dead96b7ecdf368fcd64b749bdf30e79b"
Jul 19 08:14:16.034169 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:16.034148    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9_odh-ai-gateway-infra(2d7c278a-a8ff-4a4f-839f-96cbf0d55759)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759"
Jul 19 08:14:16.050416 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:16.050389    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:14:17.175522 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:17.175495    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:14:17.261568 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:17.261534    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca\") pod \"2d7c278a-a8ff-4a4f-839f-96cbf0d55759\" (UID: \"2d7c278a-a8ff-4a4f-839f-96cbf0d55759\") "
Jul 19 08:14:17.261909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:17.261886    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d7c278a-a8ff-4a4f-839f-96cbf0d55759" (UID: "2d7c278a-a8ff-4a4f-839f-96cbf0d55759"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:14:17.362798 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:17.362766    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d7c278a-a8ff-4a4f-839f-96cbf0d55759-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:14:18.046052 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.046019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9" event={"ID":"2d7c278a-a8ff-4a4f-839f-96cbf0d55759","Type":"ContainerDied","Data":"9383f41af4ecc022a59c03c87239e85954ba3f715adc6555a0b9e6249f2576e5"}
Jul 19 08:14:18.046052 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.046054    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"
Jul 19 08:14:18.046285 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.046083    2559 scope.go:117] "RemoveContainer" containerID="76f4b8b8b5a4e53661fd0d5a7102b80dead96b7ecdf368fcd64b749bdf30e79b"
Jul 19 08:14:18.069562 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.069538    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:14:18.075364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.075341    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-4t8v9"]
Jul 19 08:14:18.460362 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:18.460329    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" path="/var/lib/kubelet/pods/2d7c278a-a8ff-4a4f-839f-96cbf0d55759/volumes"
Jul 19 08:14:34.433542 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433504    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:34.433968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433889    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.433968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433901    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.433968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433925    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.433968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433931    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.434098 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433983    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.434098 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.433994    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:14:34.442075 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.442050    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:34.443225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.443195    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:34.444769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.444746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:14:34.444863 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.444746    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:14:34.519615 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.514846    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26\" (UID: \"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:34.616173 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.616135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26\" (UID: \"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:34.616774 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.616742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26\" (UID: \"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:34.754022 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.753943    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:34.887037 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:34.887013    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:34.888507 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:14:34.888474    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d424fad_2d6f_4ee7_bde4_cffdeb1b38fc.slice/crio-28e6ca2a1e42a907a9800970ed473c04f99c8b0bda7630a60d8f1e8d127d7e0b WatchSource:0}: Error finding container 28e6ca2a1e42a907a9800970ed473c04f99c8b0bda7630a60d8f1e8d127d7e0b: Status 404 returned error can't find the container with id 28e6ca2a1e42a907a9800970ed473c04f99c8b0bda7630a60d8f1e8d127d7e0b
Jul 19 08:14:35.136002 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:35.135969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerStarted","Data":"ad8be7def87d0f0223511d011acefea2387f864084f2b34b0299c7d8291f88bc"}
Jul 19 08:14:35.136180 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:35.136022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerStarted","Data":"28e6ca2a1e42a907a9800970ed473c04f99c8b0bda7630a60d8f1e8d127d7e0b"}
Jul 19 08:14:35.150764 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:35.150683    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" podStartSLOduration=1.150667818 podStartE2EDuration="1.150667818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:14:35.149897621 +0000 UTC m=+1911.659495522" watchObservedRunningTime="2026-07-19 08:14:35.150667818 +0000 UTC m=+1911.660265752"
Jul 19 08:14:36.142639 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:36.142604    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerID="ad8be7def87d0f0223511d011acefea2387f864084f2b34b0299c7d8291f88bc" exitCode=7
Jul 19 08:14:36.143046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:36.142692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerDied","Data":"ad8be7def87d0f0223511d011acefea2387f864084f2b34b0299c7d8291f88bc"}
Jul 19 08:14:36.143046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:36.142862    2559 scope.go:117] "RemoveContainer" containerID="ad8be7def87d0f0223511d011acefea2387f864084f2b34b0299c7d8291f88bc"
Jul 19 08:14:37.149666 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:37.149622    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerID="001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae" exitCode=7
Jul 19 08:14:37.150060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:37.149693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerDied","Data":"001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae"}
Jul 19 08:14:37.150060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:37.149766    2559 scope.go:117] "RemoveContainer" containerID="ad8be7def87d0f0223511d011acefea2387f864084f2b34b0299c7d8291f88bc"
Jul 19 08:14:37.150060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:37.149993    2559 scope.go:117] "RemoveContainer" containerID="001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae"
Jul 19 08:14:37.150218 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:37.150200    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26_odh-ai-gateway-infra(6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc"
Jul 19 08:14:38.156400 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:38.156373    2559 scope.go:117] "RemoveContainer" containerID="001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae"
Jul 19 08:14:38.156802 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:38.156538    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26_odh-ai-gateway-infra(6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc"
Jul 19 08:14:50.455438 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:50.455408    2559 scope.go:117] "RemoveContainer" containerID="001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae"
Jul 19 08:14:51.225518 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:51.225490    2559 generic.go:356] "Generic (PLEG): container finished" podID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerID="9c0fb8d6dbab69e16fcaa0e0481f8b6c83dbfec7c6fdc72309e29e15f77dac54" exitCode=6
Jul 19 08:14:51.225671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:51.225569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerDied","Data":"9c0fb8d6dbab69e16fcaa0e0481f8b6c83dbfec7c6fdc72309e29e15f77dac54"}
Jul 19 08:14:51.225671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:51.225633    2559 scope.go:117] "RemoveContainer" containerID="001e73c74bf7bc673a76817ecb70bec7803c0fe8c7b89100ab6fd24df507deae"
Jul 19 08:14:51.226061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:51.226043    2559 scope.go:117] "RemoveContainer" containerID="9c0fb8d6dbab69e16fcaa0e0481f8b6c83dbfec7c6fdc72309e29e15f77dac54"
Jul 19 08:14:51.226302 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:14:51.226265    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26_odh-ai-gateway-infra(6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc"
Jul 19 08:14:52.250528 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:52.250494    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:52.374944 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:52.374918    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:52.484607 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:52.484578    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca\") pod \"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc\" (UID: \"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc\") "
Jul 19 08:14:52.484936 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:52.484914    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" (UID: "6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:14:52.585730 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:52.585688    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:14:53.239432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:53.239398    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"
Jul 19 08:14:53.239644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:53.239441    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26" event={"ID":"6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc","Type":"ContainerDied","Data":"28e6ca2a1e42a907a9800970ed473c04f99c8b0bda7630a60d8f1e8d127d7e0b"}
Jul 19 08:14:53.239644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:53.239534    2559 scope.go:117] "RemoveContainer" containerID="9c0fb8d6dbab69e16fcaa0e0481f8b6c83dbfec7c6fdc72309e29e15f77dac54"
Jul 19 08:14:53.262818 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:53.262792    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:53.266607 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:53.266580    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-mfh26"]
Jul 19 08:14:54.460677 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:14:54.460646    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" path="/var/lib/kubelet/pods/6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc/volumes"
Jul 19 08:15:00.132481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132444    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"]
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132823    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132838    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132855    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132860    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132872    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.132876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132879    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.133057 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132950    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.133057 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132959    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:00.133057 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.132966    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d7c278a-a8ff-4a4f-839f-96cbf0d55759" containerName="revoke-keys"
Jul 19 08:15:00.137472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.137456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"
Jul 19 08:15:00.140266 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.140241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:15:00.142969 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.142938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"]
Jul 19 08:15:00.149458 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.149433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"
Jul 19 08:15:00.280915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:00.280890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"]
Jul 19 08:15:00.283172 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:15:00.283146    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c92e219_7381_4ac8_bf38_b761d051b292.slice/crio-0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302 WatchSource:0}: Error finding container 0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302: Status 404 returned error can't find the container with id 0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302
Jul 19 08:15:01.290257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:01.290225    2559 generic.go:356] "Generic (PLEG): container finished" podID="7c92e219-7381-4ac8-bf38-b761d051b292" containerID="742d3d9717e05dfc53937cf15216d4ad4f27b4c55e4c7c0a5fa73564cae5ccf0" exitCode=0
Jul 19 08:15:01.290662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:01.290317    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l" event={"ID":"7c92e219-7381-4ac8-bf38-b761d051b292","Type":"ContainerDied","Data":"742d3d9717e05dfc53937cf15216d4ad4f27b4c55e4c7c0a5fa73564cae5ccf0"}
Jul 19 08:15:01.290662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:01.290385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l" event={"ID":"7c92e219-7381-4ac8-bf38-b761d051b292","Type":"ContainerStarted","Data":"0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302"}
Jul 19 08:15:02.431861 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.431837    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"
Jul 19 08:15:02.640539 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.640502    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:02.640926 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.640913    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c92e219-7381-4ac8-bf38-b761d051b292" containerName="cleanup"
Jul 19 08:15:02.640978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.640928    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c92e219-7381-4ac8-bf38-b761d051b292" containerName="cleanup"
Jul 19 08:15:02.640978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.640937    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:02.640978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.640943    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:02.641074 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.641006    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c92e219-7381-4ac8-bf38-b761d051b292" containerName="cleanup"
Jul 19 08:15:02.641074 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.641017    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d424fad-2d6f-4ee7-bde4-cffdeb1b38fc" containerName="revoke-keys"
Jul 19 08:15:02.643945 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.643926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:02.646307 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.646290    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:15:02.651525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.651457    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:02.677924 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.677900    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7\" (UID: \"1051712c-72fa-428c-9fc0-835e6735e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:02.778987 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.778957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7\" (UID: \"1051712c-72fa-428c-9fc0-835e6735e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:02.779576 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.779556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7\" (UID: \"1051712c-72fa-428c-9fc0-835e6735e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:02.956237 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:02.956156    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:03.089013 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:03.088988    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:03.091146 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:15:03.091118    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1051712c_72fa_428c_9fc0_835e6735e788.slice/crio-1f0b234357f3017360b19eada896c58834b461c94ba55e34080ade61c1d525f7 WatchSource:0}: Error finding container 1f0b234357f3017360b19eada896c58834b461c94ba55e34080ade61c1d525f7: Status 404 returned error can't find the container with id 1f0b234357f3017360b19eada896c58834b461c94ba55e34080ade61c1d525f7
Jul 19 08:15:03.303319 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:03.303290    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l"
Jul 19 08:15:03.303455 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:03.303339    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29740815-cl74l" event={"ID":"7c92e219-7381-4ac8-bf38-b761d051b292","Type":"ContainerDied","Data":"0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302"}
Jul 19 08:15:03.303455 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:03.303384    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0699bb54d218c66e47bc92882f02425b7ebcf05f12f35bbeaaff317325f09302"
Jul 19 08:15:03.304593 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:03.304568    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" event={"ID":"1051712c-72fa-428c-9fc0-835e6735e788","Type":"ContainerStarted","Data":"1f0b234357f3017360b19eada896c58834b461c94ba55e34080ade61c1d525f7"}
Jul 19 08:15:04.312344 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:04.312312    2559 generic.go:356] "Generic (PLEG): container finished" podID="1051712c-72fa-428c-9fc0-835e6735e788" containerID="9d9822990e952eaa04907c617094f4d402d8e5d863202ce14301d9ae1f072732" exitCode=7
Jul 19 08:15:04.312747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:04.312357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" event={"ID":"1051712c-72fa-428c-9fc0-835e6735e788","Type":"ContainerDied","Data":"9d9822990e952eaa04907c617094f4d402d8e5d863202ce14301d9ae1f072732"}
Jul 19 08:15:04.312747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:04.312543    2559 scope.go:117] "RemoveContainer" containerID="9d9822990e952eaa04907c617094f4d402d8e5d863202ce14301d9ae1f072732"
Jul 19 08:15:05.319681 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:05.319650    2559 generic.go:356] "Generic (PLEG): container finished" podID="1051712c-72fa-428c-9fc0-835e6735e788" containerID="e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535" exitCode=7
Jul 19 08:15:05.320116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:05.319740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" event={"ID":"1051712c-72fa-428c-9fc0-835e6735e788","Type":"ContainerDied","Data":"e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535"}
Jul 19 08:15:05.320116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:05.319805    2559 scope.go:117] "RemoveContainer" containerID="9d9822990e952eaa04907c617094f4d402d8e5d863202ce14301d9ae1f072732"
Jul 19 08:15:05.320116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:05.320000    2559 scope.go:117] "RemoveContainer" containerID="e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535"
Jul 19 08:15:05.320246 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:05.320205    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7_odh-ai-gateway-infra(1051712c-72fa-428c-9fc0-835e6735e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" podUID="1051712c-72fa-428c-9fc0-835e6735e788"
Jul 19 08:15:06.327056 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:06.327030    2559 scope.go:117] "RemoveContainer" containerID="e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535"
Jul 19 08:15:06.327444 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:06.327246    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7_odh-ai-gateway-infra(1051712c-72fa-428c-9fc0-835e6735e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" podUID="1051712c-72fa-428c-9fc0-835e6735e788"
Jul 19 08:15:19.454836 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:19.454801    2559 scope.go:117] "RemoveContainer" containerID="e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535"
Jul 19 08:15:20.408188 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:20.408099    2559 generic.go:356] "Generic (PLEG): container finished" podID="1051712c-72fa-428c-9fc0-835e6735e788" containerID="62a8354051f960bc0e3858da1746f99dc5a736872971a3598e78c14276621888" exitCode=7
Jul 19 08:15:20.408329 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:20.408185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" event={"ID":"1051712c-72fa-428c-9fc0-835e6735e788","Type":"ContainerDied","Data":"62a8354051f960bc0e3858da1746f99dc5a736872971a3598e78c14276621888"}
Jul 19 08:15:20.408329 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:20.408249    2559 scope.go:117] "RemoveContainer" containerID="e3348304ad978e9d9ffbed24c052cd44ab04dcae99f5a71b92cd05a74c80a535"
Jul 19 08:15:20.408482 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:20.408470    2559 scope.go:117] "RemoveContainer" containerID="62a8354051f960bc0e3858da1746f99dc5a736872971a3598e78c14276621888"
Jul 19 08:15:20.408724 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:20.408684    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7_odh-ai-gateway-infra(1051712c-72fa-428c-9fc0-835e6735e788)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" podUID="1051712c-72fa-428c-9fc0-835e6735e788"
Jul 19 08:15:21.431329 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:21.431287    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:21.569672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:21.569650    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:21.643461 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:21.643412    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca\") pod \"1051712c-72fa-428c-9fc0-835e6735e788\" (UID: \"1051712c-72fa-428c-9fc0-835e6735e788\") "
Jul 19 08:15:21.643806 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:21.643785    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1051712c-72fa-428c-9fc0-835e6735e788" (UID: "1051712c-72fa-428c-9fc0-835e6735e788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:15:21.744170 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:21.744088    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051712c-72fa-428c-9fc0-835e6735e788-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:15:22.422036 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.422011    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"
Jul 19 08:15:22.422209 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.422010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7" event={"ID":"1051712c-72fa-428c-9fc0-835e6735e788","Type":"ContainerDied","Data":"1f0b234357f3017360b19eada896c58834b461c94ba55e34080ade61c1d525f7"}
Jul 19 08:15:22.422209 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.422143    2559 scope.go:117] "RemoveContainer" containerID="62a8354051f960bc0e3858da1746f99dc5a736872971a3598e78c14276621888"
Jul 19 08:15:22.443228 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.443200    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:22.450920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.450897    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-nlbs7"]
Jul 19 08:15:22.466777 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:22.466750    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1051712c-72fa-428c-9fc0-835e6735e788" path="/var/lib/kubelet/pods/1051712c-72fa-428c-9fc0-835e6735e788/volumes"
Jul 19 08:15:24.474767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.474735    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475051    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475062    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475078    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475083    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475092    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475129 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475098    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475160    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475170    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.475332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.475178    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1051712c-72fa-428c-9fc0-835e6735e788" containerName="revoke-keys"
Jul 19 08:15:24.479522 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.479505    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:24.482755 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.482331    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:15:24.482755 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.482581    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:15:24.484748 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.484725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:24.571076 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.571040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5\" (UID: \"3fa7e889-8766-4809-ab24-822b2068d487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:24.671634 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.671593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5\" (UID: \"3fa7e889-8766-4809-ab24-822b2068d487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:24.672263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.672244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5\" (UID: \"3fa7e889-8766-4809-ab24-822b2068d487\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:24.791894 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.791807    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:24.921788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:24.921764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:24.923781 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:15:24.923747    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa7e889_8766_4809_ab24_822b2068d487.slice/crio-11d40fdf7fe1c6b8b80ada20589152f89a76c2ce975a481db6a339564eea1e25 WatchSource:0}: Error finding container 11d40fdf7fe1c6b8b80ada20589152f89a76c2ce975a481db6a339564eea1e25: Status 404 returned error can't find the container with id 11d40fdf7fe1c6b8b80ada20589152f89a76c2ce975a481db6a339564eea1e25
Jul 19 08:15:25.440447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:25.440358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerStarted","Data":"16cf305e8f27e770710eb2e7e1761ce50c6ecd1edae7e33b93e157457ff92321"}
Jul 19 08:15:25.440447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:25.440407    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerStarted","Data":"11d40fdf7fe1c6b8b80ada20589152f89a76c2ce975a481db6a339564eea1e25"}
Jul 19 08:15:25.455804 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:25.455757    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" podStartSLOduration=1.455743208 podStartE2EDuration="1.455743208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:15:25.454097179 +0000 UTC m=+1961.963695080" watchObservedRunningTime="2026-07-19 08:15:25.455743208 +0000 UTC m=+1961.965341104"
Jul 19 08:15:26.453599 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:26.453559    2559 generic.go:356] "Generic (PLEG): container finished" podID="3fa7e889-8766-4809-ab24-822b2068d487" containerID="16cf305e8f27e770710eb2e7e1761ce50c6ecd1edae7e33b93e157457ff92321" exitCode=6
Jul 19 08:15:26.454008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:26.453635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerDied","Data":"16cf305e8f27e770710eb2e7e1761ce50c6ecd1edae7e33b93e157457ff92321"}
Jul 19 08:15:26.454008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:26.453835    2559 scope.go:117] "RemoveContainer" containerID="16cf305e8f27e770710eb2e7e1761ce50c6ecd1edae7e33b93e157457ff92321"
Jul 19 08:15:27.461105 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:27.461013    2559 generic.go:356] "Generic (PLEG): container finished" podID="3fa7e889-8766-4809-ab24-822b2068d487" containerID="226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76" exitCode=6
Jul 19 08:15:27.461509 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:27.461099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerDied","Data":"226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76"}
Jul 19 08:15:27.461509 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:27.461148    2559 scope.go:117] "RemoveContainer" containerID="16cf305e8f27e770710eb2e7e1761ce50c6ecd1edae7e33b93e157457ff92321"
Jul 19 08:15:27.461509 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:27.461273    2559 scope.go:117] "RemoveContainer" containerID="226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76"
Jul 19 08:15:27.461509 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:27.461469    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5_odh-ai-gateway-infra(3fa7e889-8766-4809-ab24-822b2068d487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" podUID="3fa7e889-8766-4809-ab24-822b2068d487"
Jul 19 08:15:28.467632 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:28.467609    2559 scope.go:117] "RemoveContainer" containerID="226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76"
Jul 19 08:15:28.468025 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:28.467819    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5_odh-ai-gateway-infra(3fa7e889-8766-4809-ab24-822b2068d487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" podUID="3fa7e889-8766-4809-ab24-822b2068d487"
Jul 19 08:15:39.455637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:39.455605    2559 scope.go:117] "RemoveContainer" containerID="226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76"
Jul 19 08:15:40.532516 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:40.532434    2559 generic.go:356] "Generic (PLEG): container finished" podID="3fa7e889-8766-4809-ab24-822b2068d487" containerID="3dac0a2428cdcc99e3ff5c04a0876e48a4ccbac14a874a832e88b707c5f1a855" exitCode=6
Jul 19 08:15:40.532963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:40.532509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerDied","Data":"3dac0a2428cdcc99e3ff5c04a0876e48a4ccbac14a874a832e88b707c5f1a855"}
Jul 19 08:15:40.532963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:40.532569    2559 scope.go:117] "RemoveContainer" containerID="226a3e56bf1c1bda81e65263ac3294a7bd893612ab6310290307d62eb5361f76"
Jul 19 08:15:40.532963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:40.532834    2559 scope.go:117] "RemoveContainer" containerID="3dac0a2428cdcc99e3ff5c04a0876e48a4ccbac14a874a832e88b707c5f1a855"
Jul 19 08:15:40.533140 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:40.533027    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5_odh-ai-gateway-infra(3fa7e889-8766-4809-ab24-822b2068d487)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" podUID="3fa7e889-8766-4809-ab24-822b2068d487"
Jul 19 08:15:41.556857 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:41.556827    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:41.709870 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:41.709843    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:41.731766 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:41.731596    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca\") pod \"3fa7e889-8766-4809-ab24-822b2068d487\" (UID: \"3fa7e889-8766-4809-ab24-822b2068d487\") "
Jul 19 08:15:41.732312 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:41.732278    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3fa7e889-8766-4809-ab24-822b2068d487" (UID: "3fa7e889-8766-4809-ab24-822b2068d487"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:15:41.833248 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:41.833212    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa7e889-8766-4809-ab24-822b2068d487-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:15:42.545324 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:42.545295    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"
Jul 19 08:15:42.545504 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:42.545293    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5" event={"ID":"3fa7e889-8766-4809-ab24-822b2068d487","Type":"ContainerDied","Data":"11d40fdf7fe1c6b8b80ada20589152f89a76c2ce975a481db6a339564eea1e25"}
Jul 19 08:15:42.545504 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:42.545438    2559 scope.go:117] "RemoveContainer" containerID="3dac0a2428cdcc99e3ff5c04a0876e48a4ccbac14a874a832e88b707c5f1a855"
Jul 19 08:15:42.561162 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:42.561137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:42.566904 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:42.566883    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-jvkb5"]
Jul 19 08:15:44.460404 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:44.460368    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa7e889-8766-4809-ab24-822b2068d487" path="/var/lib/kubelet/pods/3fa7e889-8766-4809-ab24-822b2068d487/volumes"
Jul 19 08:15:46.792794 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.792758    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:15:46.793167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793098    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.793167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793109    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.793167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793126    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.793167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793131    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.793300 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793193    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.793300 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.793203    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:15:46.796179 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.796164    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:15:46.798376 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.798337    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:15:46.798514 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.798450    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:15:46.804411 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.803983    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:15:46.876235 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.876201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-84hk4\" (UID: \"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:15:46.976835 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.976803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-84hk4\" (UID: \"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:15:46.977432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:46.977412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-84hk4\" (UID: \"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:15:47.107989 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:47.107948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:15:47.239629 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:47.239603    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:15:47.241306 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:15:47.241280    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cd1094_9869_44d3_a7ab_71fd4fe78bfc.slice/crio-20c194ce371fc19987613bbbb0977253b8e0556984a9e8cb028a8db15e0ce0a8 WatchSource:0}: Error finding container 20c194ce371fc19987613bbbb0977253b8e0556984a9e8cb028a8db15e0ce0a8: Status 404 returned error can't find the container with id 20c194ce371fc19987613bbbb0977253b8e0556984a9e8cb028a8db15e0ce0a8
Jul 19 08:15:47.575316 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:47.575272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerStarted","Data":"a5b4b72ff7040f9d94d208ccd5a3057c10b9af3c9cba24d71b6dd4fcfd472869"}
Jul 19 08:15:47.575488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:47.575358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerStarted","Data":"20c194ce371fc19987613bbbb0977253b8e0556984a9e8cb028a8db15e0ce0a8"}
Jul 19 08:15:47.590512 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:47.590461    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" podStartSLOduration=1.590445958 podStartE2EDuration="1.590445958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:15:47.588997361 +0000 UTC m=+1984.098595260" watchObservedRunningTime="2026-07-19 08:15:47.590445958 +0000 UTC m=+1984.100043858"
Jul 19 08:15:48.582290 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:48.582252    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerID="a5b4b72ff7040f9d94d208ccd5a3057c10b9af3c9cba24d71b6dd4fcfd472869" exitCode=6
Jul 19 08:15:48.582813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:48.582334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerDied","Data":"a5b4b72ff7040f9d94d208ccd5a3057c10b9af3c9cba24d71b6dd4fcfd472869"}
Jul 19 08:15:48.582813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:48.582557    2559 scope.go:117] "RemoveContainer" containerID="a5b4b72ff7040f9d94d208ccd5a3057c10b9af3c9cba24d71b6dd4fcfd472869"
Jul 19 08:15:49.589770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:49.589736    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerID="59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1" exitCode=6
Jul 19 08:15:49.590191 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:49.589790    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerDied","Data":"59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1"}
Jul 19 08:15:49.590191 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:49.589842    2559 scope.go:117] "RemoveContainer" containerID="a5b4b72ff7040f9d94d208ccd5a3057c10b9af3c9cba24d71b6dd4fcfd472869"
Jul 19 08:15:49.590191 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:49.590002    2559 scope.go:117] "RemoveContainer" containerID="59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1"
Jul 19 08:15:49.590354 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:49.590228    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-84hk4_odh-ai-gateway-infra(b5cd1094-9869-44d3-a7ab-71fd4fe78bfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc"
Jul 19 08:15:50.597529 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:15:50.597503    2559 scope.go:117] "RemoveContainer" containerID="59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1"
Jul 19 08:15:50.598024 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:15:50.597717    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-84hk4_odh-ai-gateway-infra(b5cd1094-9869-44d3-a7ab-71fd4fe78bfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc"
Jul 19 08:16:04.458173 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:04.458142    2559 scope.go:117] "RemoveContainer" containerID="59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1"
Jul 19 08:16:05.681839 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:05.681804    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerID="0dec139aa4ddbb856ab8dd470e5583c4752291ca31ab1df468acf47dad3d5de1" exitCode=6
Jul 19 08:16:05.682225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:05.681885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerDied","Data":"0dec139aa4ddbb856ab8dd470e5583c4752291ca31ab1df468acf47dad3d5de1"}
Jul 19 08:16:05.682225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:05.681973    2559 scope.go:117] "RemoveContainer" containerID="59ba5f4bb6caffb91dee9e3ef7211cd21f0404a35dd126750c5b862d000fbfe1"
Jul 19 08:16:05.682334 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:05.682320    2559 scope.go:117] "RemoveContainer" containerID="0dec139aa4ddbb856ab8dd470e5583c4752291ca31ab1df468acf47dad3d5de1"
Jul 19 08:16:05.682530 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:16:05.682509    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-84hk4_odh-ai-gateway-infra(b5cd1094-9869-44d3-a7ab-71fd4fe78bfc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc"
Jul 19 08:16:06.707266 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:06.707230    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:16:06.840972 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:06.840950    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:16:06.957881 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:06.957800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca\") pod \"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc\" (UID: \"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc\") "
Jul 19 08:16:06.958132 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:06.958106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" (UID: "b5cd1094-9869-44d3-a7ab-71fd4fe78bfc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:16:07.059436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.059402    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:16:07.694437 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.694410    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"
Jul 19 08:16:07.694638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.694410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4" event={"ID":"b5cd1094-9869-44d3-a7ab-71fd4fe78bfc","Type":"ContainerDied","Data":"20c194ce371fc19987613bbbb0977253b8e0556984a9e8cb028a8db15e0ce0a8"}
Jul 19 08:16:07.694638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.694563    2559 scope.go:117] "RemoveContainer" containerID="0dec139aa4ddbb856ab8dd470e5583c4752291ca31ab1df468acf47dad3d5de1"
Jul 19 08:16:07.716934 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.716909    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:16:07.722364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:07.722342    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-84hk4"]
Jul 19 08:16:08.460375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:16:08.460340    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" path="/var/lib/kubelet/pods/b5cd1094-9869-44d3-a7ab-71fd4fe78bfc/volumes"
Jul 19 08:17:36.880684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.880652    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881008    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881019    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881031    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881036    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881045    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881051    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881066    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881072    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881127    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881134    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa7e889-8766-4809-ab24-822b2068d487" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881141    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.881263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.881147    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cd1094-9869-44d3-a7ab-71fd4fe78bfc" containerName="revoke-keys"
Jul 19 08:17:36.884033 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.884017    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:36.886686 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.886661    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:17:36.886810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.886735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:17:36.891248 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.891221    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:36.955951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:36.955925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz\" (UID: \"42f58389-248b-41a4-b46a-68c15c30dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:37.056965 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:37.056928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz\" (UID: \"42f58389-248b-41a4-b46a-68c15c30dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:37.057579 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:37.057558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz\" (UID: \"42f58389-248b-41a4-b46a-68c15c30dcf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:37.196227 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:37.196145    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:37.323892 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:37.323867    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:37.326513 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:17:37.326492    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f58389_248b_41a4_b46a_68c15c30dcf1.slice/crio-e7bce83212d4b24fd270d51a4031e7f2f94508e653f9671e2a48ed9ae1a83117 WatchSource:0}: Error finding container e7bce83212d4b24fd270d51a4031e7f2f94508e653f9671e2a48ed9ae1a83117: Status 404 returned error can't find the container with id e7bce83212d4b24fd270d51a4031e7f2f94508e653f9671e2a48ed9ae1a83117
Jul 19 08:17:38.163517 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:38.163488    2559 generic.go:356] "Generic (PLEG): container finished" podID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerID="db87ca6ef2fdabac411f04c0a874dc085e766163ea9c814e9fcf5534567a39ec" exitCode=6
Jul 19 08:17:38.163899 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:38.163551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" event={"ID":"42f58389-248b-41a4-b46a-68c15c30dcf1","Type":"ContainerDied","Data":"db87ca6ef2fdabac411f04c0a874dc085e766163ea9c814e9fcf5534567a39ec"}
Jul 19 08:17:38.163899 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:38.163611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" event={"ID":"42f58389-248b-41a4-b46a-68c15c30dcf1","Type":"ContainerStarted","Data":"e7bce83212d4b24fd270d51a4031e7f2f94508e653f9671e2a48ed9ae1a83117"}
Jul 19 08:17:38.163899 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:38.163844    2559 scope.go:117] "RemoveContainer" containerID="db87ca6ef2fdabac411f04c0a874dc085e766163ea9c814e9fcf5534567a39ec"
Jul 19 08:17:39.171127 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:39.171045    2559 generic.go:356] "Generic (PLEG): container finished" podID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerID="624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98" exitCode=6
Jul 19 08:17:39.171127 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:39.171104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" event={"ID":"42f58389-248b-41a4-b46a-68c15c30dcf1","Type":"ContainerDied","Data":"624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98"}
Jul 19 08:17:39.171599 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:39.171170    2559 scope.go:117] "RemoveContainer" containerID="db87ca6ef2fdabac411f04c0a874dc085e766163ea9c814e9fcf5534567a39ec"
Jul 19 08:17:39.171599 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:39.171368    2559 scope.go:117] "RemoveContainer" containerID="624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98"
Jul 19 08:17:39.171684 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:17:39.171612    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz_odh-ai-gateway-infra(42f58389-248b-41a4-b46a-68c15c30dcf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1"
Jul 19 08:17:40.184848 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:40.184819    2559 scope.go:117] "RemoveContainer" containerID="624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98"
Jul 19 08:17:40.185241 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:17:40.185009    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz_odh-ai-gateway-infra(42f58389-248b-41a4-b46a-68c15c30dcf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1"
Jul 19 08:17:54.458151 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:54.458124    2559 scope.go:117] "RemoveContainer" containerID="624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98"
Jul 19 08:17:55.265790 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:55.265753    2559 generic.go:356] "Generic (PLEG): container finished" podID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerID="fd543d90ca65f83c3ded27d32eeca9e1948df978a88d39e848ccef02bcfe945f" exitCode=6
Jul 19 08:17:55.265955 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:55.265831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" event={"ID":"42f58389-248b-41a4-b46a-68c15c30dcf1","Type":"ContainerDied","Data":"fd543d90ca65f83c3ded27d32eeca9e1948df978a88d39e848ccef02bcfe945f"}
Jul 19 08:17:55.265955 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:55.265889    2559 scope.go:117] "RemoveContainer" containerID="624f5777c55145c9db68b4d558fb9511b3c9a99d49413f898167c88a4e761d98"
Jul 19 08:17:55.266183 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:55.266166    2559 scope.go:117] "RemoveContainer" containerID="fd543d90ca65f83c3ded27d32eeca9e1948df978a88d39e848ccef02bcfe945f"
Jul 19 08:17:55.266389 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:17:55.266371    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz_odh-ai-gateway-infra(42f58389-248b-41a4-b46a-68c15c30dcf1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1"
Jul 19 08:17:56.290128 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:56.290075    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:56.422347 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:56.422324    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:56.538481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:56.538448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca\") pod \"42f58389-248b-41a4-b46a-68c15c30dcf1\" (UID: \"42f58389-248b-41a4-b46a-68c15c30dcf1\") "
Jul 19 08:17:56.538963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:56.538839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42f58389-248b-41a4-b46a-68c15c30dcf1" (UID: "42f58389-248b-41a4-b46a-68c15c30dcf1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:17:56.539360 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:56.539343    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42f58389-248b-41a4-b46a-68c15c30dcf1-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:17:57.279481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:57.279456    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"
Jul 19 08:17:57.279657 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:57.279500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz" event={"ID":"42f58389-248b-41a4-b46a-68c15c30dcf1","Type":"ContainerDied","Data":"e7bce83212d4b24fd270d51a4031e7f2f94508e653f9671e2a48ed9ae1a83117"}
Jul 19 08:17:57.279657 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:57.279556    2559 scope.go:117] "RemoveContainer" containerID="fd543d90ca65f83c3ded27d32eeca9e1948df978a88d39e848ccef02bcfe945f"
Jul 19 08:17:57.302079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:57.302045    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:57.307753 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:57.307730    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-qvzfz"]
Jul 19 08:17:58.461201 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:17:58.461164    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" path="/var/lib/kubelet/pods/42f58389-248b-41a4-b46a-68c15c30dcf1/volumes"
Jul 19 08:18:14.616258 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616221    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616565    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616576    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616592    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616597    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616651    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616658    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.616773 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.616666    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:14.622522 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.622489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:14.626310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.626283    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:18:14.626631 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.626582    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:18:14.627557 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.627530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:14.694258 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.694217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr\" (UID: \"6f911755-b09b-4b0f-8ee3-570aaefc6cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:14.795335 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.795292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr\" (UID: \"6f911755-b09b-4b0f-8ee3-570aaefc6cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:14.795925 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.795907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr\" (UID: \"6f911755-b09b-4b0f-8ee3-570aaefc6cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:14.937470 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:14.937391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:15.071890 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:15.071867    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:15.073890 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:18:15.073859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f911755_b09b_4b0f_8ee3_570aaefc6cca.slice/crio-2905060d5d394fb6a6a942d959f9a64d58c0cca32bdcdeb2619da12dc54ba7e1 WatchSource:0}: Error finding container 2905060d5d394fb6a6a942d959f9a64d58c0cca32bdcdeb2619da12dc54ba7e1: Status 404 returned error can't find the container with id 2905060d5d394fb6a6a942d959f9a64d58c0cca32bdcdeb2619da12dc54ba7e1
Jul 19 08:18:15.381838 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:15.381801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerStarted","Data":"ca27828e2851a63df64138dffeb8efd701fc4f52a03b9f3880d469374e86f9fd"}
Jul 19 08:18:15.382009 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:15.381860    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerStarted","Data":"2905060d5d394fb6a6a942d959f9a64d58c0cca32bdcdeb2619da12dc54ba7e1"}
Jul 19 08:18:15.397141 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:15.397077    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" podStartSLOduration=1.397061708 podStartE2EDuration="1.397061708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:18:15.396070324 +0000 UTC m=+2131.905668223" watchObservedRunningTime="2026-07-19 08:18:15.397061708 +0000 UTC m=+2131.906659663"
Jul 19 08:18:16.389224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:16.389184    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerID="ca27828e2851a63df64138dffeb8efd701fc4f52a03b9f3880d469374e86f9fd" exitCode=6
Jul 19 08:18:16.389621 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:16.389265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerDied","Data":"ca27828e2851a63df64138dffeb8efd701fc4f52a03b9f3880d469374e86f9fd"}
Jul 19 08:18:16.389621 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:16.389443    2559 scope.go:117] "RemoveContainer" containerID="ca27828e2851a63df64138dffeb8efd701fc4f52a03b9f3880d469374e86f9fd"
Jul 19 08:18:17.396714 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:17.396608    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerID="ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05" exitCode=6
Jul 19 08:18:17.397088 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:17.396716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerDied","Data":"ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05"}
Jul 19 08:18:17.397088 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:17.396781    2559 scope.go:117] "RemoveContainer" containerID="ca27828e2851a63df64138dffeb8efd701fc4f52a03b9f3880d469374e86f9fd"
Jul 19 08:18:17.397088 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:17.397016    2559 scope.go:117] "RemoveContainer" containerID="ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05"
Jul 19 08:18:17.397253 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:17.397236    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr_odh-ai-gateway-infra(6f911755-b09b-4b0f-8ee3-570aaefc6cca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca"
Jul 19 08:18:18.404899 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:18.404872    2559 scope.go:117] "RemoveContainer" containerID="ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05"
Jul 19 08:18:18.405371 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:18.405097    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr_odh-ai-gateway-infra(6f911755-b09b-4b0f-8ee3-570aaefc6cca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca"
Jul 19 08:18:32.460511 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:32.460480    2559 scope.go:117] "RemoveContainer" containerID="ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05"
Jul 19 08:18:33.492708 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:33.492611    2559 generic.go:356] "Generic (PLEG): container finished" podID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerID="1c6fa4abcd9c6c24645fad8c5226e131a30b87f9989bc6767f2491c39d2eb80f" exitCode=6
Jul 19 08:18:33.492708 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:33.492659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerDied","Data":"1c6fa4abcd9c6c24645fad8c5226e131a30b87f9989bc6767f2491c39d2eb80f"}
Jul 19 08:18:33.493199 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:33.492729    2559 scope.go:117] "RemoveContainer" containerID="ae20723512f40500974f31ccf571f94808c4a32505f841fe0ccadf876d262d05"
Jul 19 08:18:33.493199 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:33.493078    2559 scope.go:117] "RemoveContainer" containerID="1c6fa4abcd9c6c24645fad8c5226e131a30b87f9989bc6767f2491c39d2eb80f"
Jul 19 08:18:33.493327 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:33.493304    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr_odh-ai-gateway-infra(6f911755-b09b-4b0f-8ee3-570aaefc6cca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca"
Jul 19 08:18:34.517521 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:34.517486    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:34.653171 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:34.653141    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:34.775619 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:34.775528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca\") pod \"6f911755-b09b-4b0f-8ee3-570aaefc6cca\" (UID: \"6f911755-b09b-4b0f-8ee3-570aaefc6cca\") "
Jul 19 08:18:34.775948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:34.775924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f911755-b09b-4b0f-8ee3-570aaefc6cca" (UID: "6f911755-b09b-4b0f-8ee3-570aaefc6cca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:18:34.877070 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:34.877030    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f911755-b09b-4b0f-8ee3-570aaefc6cca-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:18:35.506001 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:35.505958    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"
Jul 19 08:18:35.506001 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:35.505979    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr" event={"ID":"6f911755-b09b-4b0f-8ee3-570aaefc6cca","Type":"ContainerDied","Data":"2905060d5d394fb6a6a942d959f9a64d58c0cca32bdcdeb2619da12dc54ba7e1"}
Jul 19 08:18:35.506232 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:35.506043    2559 scope.go:117] "RemoveContainer" containerID="1c6fa4abcd9c6c24645fad8c5226e131a30b87f9989bc6767f2491c39d2eb80f"
Jul 19 08:18:35.528742 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:35.528720    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:35.534289 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:35.534266    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-d5mqr"]
Jul 19 08:18:36.460953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.460920    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" path="/var/lib/kubelet/pods/6f911755-b09b-4b0f-8ee3-570aaefc6cca/volumes"
Jul 19 08:18:36.928265 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928230    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928568    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928578    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928593    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928607    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928612    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928621    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:36.928638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928626    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f58389-248b-41a4-b46a-68c15c30dcf1" containerName="revoke-keys"
Jul 19 08:18:36.928917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928678    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.928917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.928687    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:18:36.933109 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.933092    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:36.935609 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.935570    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:18:36.935779 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.935747    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:18:36.937640 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.937619    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:36.995560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:36.995529    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l85gf\" (UID: \"ea6174ef-442f-4103-92de-810fdbb1fc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:37.096829 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:37.096789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l85gf\" (UID: \"ea6174ef-442f-4103-92de-810fdbb1fc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:37.097422 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:37.097402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-l85gf\" (UID: \"ea6174ef-442f-4103-92de-810fdbb1fc75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:37.244864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:37.244774    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:37.373311 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:37.373289    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:37.375802 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:18:37.375774    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6174ef_442f_4103_92de_810fdbb1fc75.slice/crio-b625a87ce6915ae52d354815d198341fdc7206fe01b6e75a7574f0b9871f2188 WatchSource:0}: Error finding container b625a87ce6915ae52d354815d198341fdc7206fe01b6e75a7574f0b9871f2188: Status 404 returned error can't find the container with id b625a87ce6915ae52d354815d198341fdc7206fe01b6e75a7574f0b9871f2188
Jul 19 08:18:37.518363 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:37.518266    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" event={"ID":"ea6174ef-442f-4103-92de-810fdbb1fc75","Type":"ContainerStarted","Data":"b625a87ce6915ae52d354815d198341fdc7206fe01b6e75a7574f0b9871f2188"}
Jul 19 08:18:38.525211 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:38.525176    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerID="e6d8203121c5ebac60f4d71113f97214666376c4ce71e6f63133b9f927518984" exitCode=6
Jul 19 08:18:38.525596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:38.525257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" event={"ID":"ea6174ef-442f-4103-92de-810fdbb1fc75","Type":"ContainerDied","Data":"e6d8203121c5ebac60f4d71113f97214666376c4ce71e6f63133b9f927518984"}
Jul 19 08:18:38.525596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:38.525391    2559 scope.go:117] "RemoveContainer" containerID="e6d8203121c5ebac60f4d71113f97214666376c4ce71e6f63133b9f927518984"
Jul 19 08:18:39.531907 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:39.531812    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerID="d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d" exitCode=6
Jul 19 08:18:39.532342 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:39.531900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" event={"ID":"ea6174ef-442f-4103-92de-810fdbb1fc75","Type":"ContainerDied","Data":"d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d"}
Jul 19 08:18:39.532342 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:39.531996    2559 scope.go:117] "RemoveContainer" containerID="e6d8203121c5ebac60f4d71113f97214666376c4ce71e6f63133b9f927518984"
Jul 19 08:18:39.532342 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:39.532139    2559 scope.go:117] "RemoveContainer" containerID="d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d"
Jul 19 08:18:39.532470 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:39.532374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l85gf_odh-ai-gateway-infra(ea6174ef-442f-4103-92de-810fdbb1fc75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75"
Jul 19 08:18:40.539045 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:40.539019    2559 scope.go:117] "RemoveContainer" containerID="d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d"
Jul 19 08:18:40.539431 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:40.539206    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l85gf_odh-ai-gateway-infra(ea6174ef-442f-4103-92de-810fdbb1fc75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75"
Jul 19 08:18:54.457920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:54.457894    2559 scope.go:117] "RemoveContainer" containerID="d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d"
Jul 19 08:18:55.620263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:55.620229    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerID="95c9a1ea1942cc0be04c806f4792c4163c70a1b74f8681da9ddf5d89a6b8b077" exitCode=6
Jul 19 08:18:55.620688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:55.620290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" event={"ID":"ea6174ef-442f-4103-92de-810fdbb1fc75","Type":"ContainerDied","Data":"95c9a1ea1942cc0be04c806f4792c4163c70a1b74f8681da9ddf5d89a6b8b077"}
Jul 19 08:18:55.620688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:55.620347    2559 scope.go:117] "RemoveContainer" containerID="d0c8ed5bbb3c3568dacb42b985f0bd8c2b75bfcf6013b22a5c0f2444b466393d"
Jul 19 08:18:55.620688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:55.620588    2559 scope.go:117] "RemoveContainer" containerID="95c9a1ea1942cc0be04c806f4792c4163c70a1b74f8681da9ddf5d89a6b8b077"
Jul 19 08:18:55.620860 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:18:55.620840    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-l85gf_odh-ai-gateway-infra(ea6174ef-442f-4103-92de-810fdbb1fc75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75"
Jul 19 08:18:56.644671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:56.644636    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:56.783959 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:56.783928    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:56.879141 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:56.879107    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca\") pod \"ea6174ef-442f-4103-92de-810fdbb1fc75\" (UID: \"ea6174ef-442f-4103-92de-810fdbb1fc75\") "
Jul 19 08:18:56.879474 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:56.879452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea6174ef-442f-4103-92de-810fdbb1fc75" (UID: "ea6174ef-442f-4103-92de-810fdbb1fc75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:18:56.980529 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:56.980446    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea6174ef-442f-4103-92de-810fdbb1fc75-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:18:57.633330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:57.633299    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"
Jul 19 08:18:57.633542 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:57.633343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf" event={"ID":"ea6174ef-442f-4103-92de-810fdbb1fc75","Type":"ContainerDied","Data":"b625a87ce6915ae52d354815d198341fdc7206fe01b6e75a7574f0b9871f2188"}
Jul 19 08:18:57.633542 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:57.633407    2559 scope.go:117] "RemoveContainer" containerID="95c9a1ea1942cc0be04c806f4792c4163c70a1b74f8681da9ddf5d89a6b8b077"
Jul 19 08:18:57.662844 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:57.662815    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:57.667258 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:57.667233    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-l85gf"]
Jul 19 08:18:58.461374 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:18:58.461342    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" path="/var/lib/kubelet/pods/ea6174ef-442f-4103-92de-810fdbb1fc75/volumes"
Jul 19 08:19:14.662458 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662770    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662782    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662794    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662800    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662870    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f911755-b09b-4b0f-8ee3-570aaefc6cca" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662877    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662883    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.662961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.662890    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:14.667107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.667090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:14.670203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.670172    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:19:14.670402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.670177    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:19:14.672770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.672746    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:14.724753 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.724723    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs\" (UID: \"c41a09de-00e4-4ff7-9e4b-222fc0718a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:14.825782 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.825751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs\" (UID: \"c41a09de-00e4-4ff7-9e4b-222fc0718a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:14.826374 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.826352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs\" (UID: \"c41a09de-00e4-4ff7-9e4b-222fc0718a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:14.979792 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:14.979689    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:15.110848 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:15.110817    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:15.112754 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:19:15.112729    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41a09de_00e4_4ff7_9e4b_222fc0718a98.slice/crio-02a63bbe66c2649fb2063386616aa9464545030680d12f74e705584d0bbe927c WatchSource:0}: Error finding container 02a63bbe66c2649fb2063386616aa9464545030680d12f74e705584d0bbe927c: Status 404 returned error can't find the container with id 02a63bbe66c2649fb2063386616aa9464545030680d12f74e705584d0bbe927c
Jul 19 08:19:15.732330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:15.732300    2559 generic.go:356] "Generic (PLEG): container finished" podID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerID="8620bb14951ce8f8f4f07c18b2016cbd428c4b1bced8f42a2a9d464c4d0c0d77" exitCode=6
Jul 19 08:19:15.732661 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:15.732376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerDied","Data":"8620bb14951ce8f8f4f07c18b2016cbd428c4b1bced8f42a2a9d464c4d0c0d77"}
Jul 19 08:19:15.732661 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:15.732425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerStarted","Data":"02a63bbe66c2649fb2063386616aa9464545030680d12f74e705584d0bbe927c"}
Jul 19 08:19:15.732661 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:15.732580    2559 scope.go:117] "RemoveContainer" containerID="8620bb14951ce8f8f4f07c18b2016cbd428c4b1bced8f42a2a9d464c4d0c0d77"
Jul 19 08:19:16.739696 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:16.739611    2559 generic.go:356] "Generic (PLEG): container finished" podID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerID="042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936" exitCode=6
Jul 19 08:19:16.740077 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:16.739720    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerDied","Data":"042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936"}
Jul 19 08:19:16.740573 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:16.739779    2559 scope.go:117] "RemoveContainer" containerID="8620bb14951ce8f8f4f07c18b2016cbd428c4b1bced8f42a2a9d464c4d0c0d77"
Jul 19 08:19:16.740673 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:16.740655    2559 scope.go:117] "RemoveContainer" containerID="042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936"
Jul 19 08:19:16.740928 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:16.740895    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs_odh-ai-gateway-infra(c41a09de-00e4-4ff7-9e4b-222fc0718a98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98"
Jul 19 08:19:17.747023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:17.746994    2559 scope.go:117] "RemoveContainer" containerID="042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936"
Jul 19 08:19:17.747441 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:17.747199    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs_odh-ai-gateway-infra(c41a09de-00e4-4ff7-9e4b-222fc0718a98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98"
Jul 19 08:19:28.455553 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:28.455517    2559 scope.go:117] "RemoveContainer" containerID="042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936"
Jul 19 08:19:28.804095 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:28.804012    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerStarted","Data":"cb07f1145e65b07b8f832bb05ee5d8cfdc546439395665664f6204b212f789f1"}
Jul 19 08:19:28.820736 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:28.820679    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" podStartSLOduration=14.820664091 podStartE2EDuration="14.820664091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:19:28.817883468 +0000 UTC m=+2205.327481367" watchObservedRunningTime="2026-07-19 08:19:28.820664091 +0000 UTC m=+2205.330261989"
Jul 19 08:19:29.810468 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:29.810433    2559 generic.go:356] "Generic (PLEG): container finished" podID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerID="cb07f1145e65b07b8f832bb05ee5d8cfdc546439395665664f6204b212f789f1" exitCode=6
Jul 19 08:19:29.810978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:29.810512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerDied","Data":"cb07f1145e65b07b8f832bb05ee5d8cfdc546439395665664f6204b212f789f1"}
Jul 19 08:19:29.810978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:29.810588    2559 scope.go:117] "RemoveContainer" containerID="042b2ae221dba980e052bb3caba2ff631c67b2b15ca922d5fff08a666a4ba936"
Jul 19 08:19:29.810978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:29.810920    2559 scope.go:117] "RemoveContainer" containerID="cb07f1145e65b07b8f832bb05ee5d8cfdc546439395665664f6204b212f789f1"
Jul 19 08:19:29.811149 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:29.811126    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs_odh-ai-gateway-infra(c41a09de-00e4-4ff7-9e4b-222fc0718a98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98"
Jul 19 08:19:29.830655 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:29.830629    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:30.951747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:30.951722    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:31.062547 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.062516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca\") pod \"c41a09de-00e4-4ff7-9e4b-222fc0718a98\" (UID: \"c41a09de-00e4-4ff7-9e4b-222fc0718a98\") "
Jul 19 08:19:31.062877 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.062856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c41a09de-00e4-4ff7-9e4b-222fc0718a98" (UID: "c41a09de-00e4-4ff7-9e4b-222fc0718a98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:19:31.164137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.164066    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c41a09de-00e4-4ff7-9e4b-222fc0718a98-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:19:31.824693 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.824668    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"
Jul 19 08:19:31.824884 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.824694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs" event={"ID":"c41a09de-00e4-4ff7-9e4b-222fc0718a98","Type":"ContainerDied","Data":"02a63bbe66c2649fb2063386616aa9464545030680d12f74e705584d0bbe927c"}
Jul 19 08:19:31.824884 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.824776    2559 scope.go:117] "RemoveContainer" containerID="cb07f1145e65b07b8f832bb05ee5d8cfdc546439395665664f6204b212f789f1"
Jul 19 08:19:31.850403 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.850376    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:31.852501 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:31.852483    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-xm6zs"]
Jul 19 08:19:32.460897 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:32.460867    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" path="/var/lib/kubelet/pods/c41a09de-00e4-4ff7-9e4b-222fc0718a98/volumes"
Jul 19 08:19:36.975324 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975287    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975648    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975660    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975676    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975681    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6174ef-442f-4103-92de-810fdbb1fc75" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975692    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975712    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975782    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975790    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.975796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.975796    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:19:36.980267 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.980252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:36.983343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.983319    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:19:36.983343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.983318    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:19:36.984966 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:36.984941    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:37.124898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.124845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg\" (UID: \"a87c0398-07f3-4c5d-98b8-831fba763087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:37.226455 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.226361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg\" (UID: \"a87c0398-07f3-4c5d-98b8-831fba763087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:37.227193 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.227169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg\" (UID: \"a87c0398-07f3-4c5d-98b8-831fba763087\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:37.291854 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.291820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:37.424917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.424890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:37.426565 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:19:37.426539    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87c0398_07f3_4c5d_98b8_831fba763087.slice/crio-a0dc52e0b64510c7544436a6558d0496aeac03f560ff526e6dd98c3d1c1681c4 WatchSource:0}: Error finding container a0dc52e0b64510c7544436a6558d0496aeac03f560ff526e6dd98c3d1c1681c4: Status 404 returned error can't find the container with id a0dc52e0b64510c7544436a6558d0496aeac03f560ff526e6dd98c3d1c1681c4
Jul 19 08:19:37.858513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.858475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerStarted","Data":"ec45f1785ae9a98291265dadfa9df53021e2a904a605b6a5dde41f0ff21feea4"}
Jul 19 08:19:37.858681 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.858529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerStarted","Data":"a0dc52e0b64510c7544436a6558d0496aeac03f560ff526e6dd98c3d1c1681c4"}
Jul 19 08:19:37.873237 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:37.873175    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" podStartSLOduration=1.873160177 podStartE2EDuration="1.873160177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:19:37.872354134 +0000 UTC m=+2214.381952033" watchObservedRunningTime="2026-07-19 08:19:37.873160177 +0000 UTC m=+2214.382758076"
Jul 19 08:19:38.865267 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:38.865231    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87c0398-07f3-4c5d-98b8-831fba763087" containerID="ec45f1785ae9a98291265dadfa9df53021e2a904a605b6a5dde41f0ff21feea4" exitCode=6
Jul 19 08:19:38.865644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:38.865307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerDied","Data":"ec45f1785ae9a98291265dadfa9df53021e2a904a605b6a5dde41f0ff21feea4"}
Jul 19 08:19:38.865644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:38.865521    2559 scope.go:117] "RemoveContainer" containerID="ec45f1785ae9a98291265dadfa9df53021e2a904a605b6a5dde41f0ff21feea4"
Jul 19 08:19:39.873079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:39.873046    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87c0398-07f3-4c5d-98b8-831fba763087" containerID="0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a" exitCode=6
Jul 19 08:19:39.873450 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:39.873131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerDied","Data":"0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a"}
Jul 19 08:19:39.873450 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:39.873189    2559 scope.go:117] "RemoveContainer" containerID="ec45f1785ae9a98291265dadfa9df53021e2a904a605b6a5dde41f0ff21feea4"
Jul 19 08:19:39.873450 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:39.873436    2559 scope.go:117] "RemoveContainer" containerID="0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a"
Jul 19 08:19:39.873719 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:39.873682    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg_odh-ai-gateway-infra(a87c0398-07f3-4c5d-98b8-831fba763087)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" podUID="a87c0398-07f3-4c5d-98b8-831fba763087"
Jul 19 08:19:40.880810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:40.880775    2559 scope.go:117] "RemoveContainer" containerID="0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a"
Jul 19 08:19:40.882729 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:40.881796    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg_odh-ai-gateway-infra(a87c0398-07f3-4c5d-98b8-831fba763087)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" podUID="a87c0398-07f3-4c5d-98b8-831fba763087"
Jul 19 08:19:54.457960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:54.457922    2559 scope.go:117] "RemoveContainer" containerID="0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a"
Jul 19 08:19:54.956684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:54.956645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerStarted","Data":"3365ea14851479396c41186812372145a7e383d0eebd39a6da06dd18aa00fbe0"}
Jul 19 08:19:55.963328 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:55.963291    2559 generic.go:356] "Generic (PLEG): container finished" podID="a87c0398-07f3-4c5d-98b8-831fba763087" containerID="3365ea14851479396c41186812372145a7e383d0eebd39a6da06dd18aa00fbe0" exitCode=6
Jul 19 08:19:55.963780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:55.963367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerDied","Data":"3365ea14851479396c41186812372145a7e383d0eebd39a6da06dd18aa00fbe0"}
Jul 19 08:19:55.963780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:55.963428    2559 scope.go:117] "RemoveContainer" containerID="0eec2bd677e22795891032e1a7ed3667bf41e76492702ab1f116dbff88e4762a"
Jul 19 08:19:55.963780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:55.963650    2559 scope.go:117] "RemoveContainer" containerID="3365ea14851479396c41186812372145a7e383d0eebd39a6da06dd18aa00fbe0"
Jul 19 08:19:55.963948 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:19:55.963929    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg_odh-ai-gateway-infra(a87c0398-07f3-4c5d-98b8-831fba763087)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" podUID="a87c0398-07f3-4c5d-98b8-831fba763087"
Jul 19 08:19:55.979561 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:55.979540    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:57.104475 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.104446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:57.210921 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.210884    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca\") pod \"a87c0398-07f3-4c5d-98b8-831fba763087\" (UID: \"a87c0398-07f3-4c5d-98b8-831fba763087\") "
Jul 19 08:19:57.211245 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.211221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a87c0398-07f3-4c5d-98b8-831fba763087" (UID: "a87c0398-07f3-4c5d-98b8-831fba763087"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:19:57.311749 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.311724    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87c0398-07f3-4c5d-98b8-831fba763087-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:19:57.976396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.976369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"
Jul 19 08:19:57.976396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.976379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg" event={"ID":"a87c0398-07f3-4c5d-98b8-831fba763087","Type":"ContainerDied","Data":"a0dc52e0b64510c7544436a6558d0496aeac03f560ff526e6dd98c3d1c1681c4"}
Jul 19 08:19:57.976630 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.976436    2559 scope.go:117] "RemoveContainer" containerID="3365ea14851479396c41186812372145a7e383d0eebd39a6da06dd18aa00fbe0"
Jul 19 08:19:57.999016 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:57.998988    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:58.004813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:58.004775    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-2k4vg"]
Jul 19 08:19:58.460962 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:19:58.460931    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" path="/var/lib/kubelet/pods/a87c0398-07f3-4c5d-98b8-831fba763087/volumes"
Jul 19 08:20:54.746084 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746048    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746387    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746397    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746410    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746415    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746432    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:20:54.746481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746438    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41a09de-00e4-4ff7-9e4b-222fc0718a98" containerName="revoke-keys"
Jul 19 08:20:54.746675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746489    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.746675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.746499    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:20:54.749427 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.749411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:20:54.751975 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.751951    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:20:54.752108 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.751981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:20:54.757630 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.757101    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:20:54.827461 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.827428    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs\" (UID: \"6eec5204-796a-4ad6-a21d-7b10b72715cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:20:54.928832 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.928799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs\" (UID: \"6eec5204-796a-4ad6-a21d-7b10b72715cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:20:54.929429 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:54.929405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs\" (UID: \"6eec5204-796a-4ad6-a21d-7b10b72715cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:20:55.061196 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:55.061173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:20:55.190888 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:55.190861    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:20:55.193476 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:20:55.193446    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eec5204_796a_4ad6_a21d_7b10b72715cf.slice/crio-36bc313947cd22ff6529ae1e1c2c39d59bd07194a002e21970e50496c982f6b0 WatchSource:0}: Error finding container 36bc313947cd22ff6529ae1e1c2c39d59bd07194a002e21970e50496c982f6b0: Status 404 returned error can't find the container with id 36bc313947cd22ff6529ae1e1c2c39d59bd07194a002e21970e50496c982f6b0
Jul 19 08:20:55.286584 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:55.286555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" event={"ID":"6eec5204-796a-4ad6-a21d-7b10b72715cf","Type":"ContainerStarted","Data":"36bc313947cd22ff6529ae1e1c2c39d59bd07194a002e21970e50496c982f6b0"}
Jul 19 08:20:56.299445 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:56.299411    2559 generic.go:356] "Generic (PLEG): container finished" podID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerID="9b2031898b7d50de7971d7c6d262c771c66876df757f78f9dfca8d47dda0eb4d" exitCode=6
Jul 19 08:20:56.299868 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:56.299471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" event={"ID":"6eec5204-796a-4ad6-a21d-7b10b72715cf","Type":"ContainerDied","Data":"9b2031898b7d50de7971d7c6d262c771c66876df757f78f9dfca8d47dda0eb4d"}
Jul 19 08:20:56.299868 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:56.299663    2559 scope.go:117] "RemoveContainer" containerID="9b2031898b7d50de7971d7c6d262c771c66876df757f78f9dfca8d47dda0eb4d"
Jul 19 08:20:57.306929 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:57.306842    2559 generic.go:356] "Generic (PLEG): container finished" podID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerID="2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9" exitCode=6
Jul 19 08:20:57.307336 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:57.306931    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" event={"ID":"6eec5204-796a-4ad6-a21d-7b10b72715cf","Type":"ContainerDied","Data":"2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9"}
Jul 19 08:20:57.307336 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:57.306994    2559 scope.go:117] "RemoveContainer" containerID="9b2031898b7d50de7971d7c6d262c771c66876df757f78f9dfca8d47dda0eb4d"
Jul 19 08:20:57.307336 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:57.307124    2559 scope.go:117] "RemoveContainer" containerID="2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9"
Jul 19 08:20:57.307462 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:20:57.307382    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs_odh-ai-gateway-infra(6eec5204-796a-4ad6-a21d-7b10b72715cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf"
Jul 19 08:20:58.315003 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:20:58.314965    2559 scope.go:117] "RemoveContainer" containerID="2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9"
Jul 19 08:20:58.315407 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:20:58.315148    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs_odh-ai-gateway-infra(6eec5204-796a-4ad6-a21d-7b10b72715cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf"
Jul 19 08:21:11.455456 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:11.455420    2559 scope.go:117] "RemoveContainer" containerID="2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9"
Jul 19 08:21:12.388491 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:12.388402    2559 generic.go:356] "Generic (PLEG): container finished" podID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerID="ff8b5bc0baae6c3a18a42365fac16bbe8b23fed4b552f7755a702e4eed50f2d0" exitCode=6
Jul 19 08:21:12.388491 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:12.388480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" event={"ID":"6eec5204-796a-4ad6-a21d-7b10b72715cf","Type":"ContainerDied","Data":"ff8b5bc0baae6c3a18a42365fac16bbe8b23fed4b552f7755a702e4eed50f2d0"}
Jul 19 08:21:12.388677 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:12.388539    2559 scope.go:117] "RemoveContainer" containerID="2191d21ab76d28f45daac305d383a27a105534c89774daf39650e6c4cf0919b9"
Jul 19 08:21:12.388936 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:12.388920    2559 scope.go:117] "RemoveContainer" containerID="ff8b5bc0baae6c3a18a42365fac16bbe8b23fed4b552f7755a702e4eed50f2d0"
Jul 19 08:21:12.389154 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:21:12.389131    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs_odh-ai-gateway-infra(6eec5204-796a-4ad6-a21d-7b10b72715cf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf"
Jul 19 08:21:13.414128 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:13.414095    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:21:13.547744 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:13.547718    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:21:13.697067 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:13.696963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca\") pod \"6eec5204-796a-4ad6-a21d-7b10b72715cf\" (UID: \"6eec5204-796a-4ad6-a21d-7b10b72715cf\") "
Jul 19 08:21:13.697376 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:13.697355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6eec5204-796a-4ad6-a21d-7b10b72715cf" (UID: "6eec5204-796a-4ad6-a21d-7b10b72715cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:21:13.797724 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:13.797648    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eec5204-796a-4ad6-a21d-7b10b72715cf-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:21:14.401056 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.401028    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"
Jul 19 08:21:14.401257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.401078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs" event={"ID":"6eec5204-796a-4ad6-a21d-7b10b72715cf","Type":"ContainerDied","Data":"36bc313947cd22ff6529ae1e1c2c39d59bd07194a002e21970e50496c982f6b0"}
Jul 19 08:21:14.401257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.401141    2559 scope.go:117] "RemoveContainer" containerID="ff8b5bc0baae6c3a18a42365fac16bbe8b23fed4b552f7755a702e4eed50f2d0"
Jul 19 08:21:14.425822 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.425794    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:21:14.438804 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.435366    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-spgxs"]
Jul 19 08:21:14.460241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:14.460210    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" path="/var/lib/kubelet/pods/6eec5204-796a-4ad6-a21d-7b10b72715cf/volumes"
Jul 19 08:21:19.828348 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828315    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828672    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828685    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828711    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828720    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828737    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828754 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828746    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828825    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828839    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:21:19.828953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.828850    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87c0398-07f3-4c5d-98b8-831fba763087" containerName="revoke-keys"
Jul 19 08:21:19.831774 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.831758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:19.834092 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.834068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:21:19.834203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.834115    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:21:19.840623 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.840592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:19.953288 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:19.953252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb\" (UID: \"1fa19f2c-de36-4cc9-b208-13c6866f7701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:20.053906 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:20.053858    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb\" (UID: \"1fa19f2c-de36-4cc9-b208-13c6866f7701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:20.054526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:20.054506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb\" (UID: \"1fa19f2c-de36-4cc9-b208-13c6866f7701\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:20.144238 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:20.144128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:20.280684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:20.280661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:20.282560 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:21:20.282536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa19f2c_de36_4cc9_b208_13c6866f7701.slice/crio-f05ab77faec5f9f9b4589ef259f2decf58cd4ac0b67d4da4549fbf2f6db18faa WatchSource:0}: Error finding container f05ab77faec5f9f9b4589ef259f2decf58cd4ac0b67d4da4549fbf2f6db18faa: Status 404 returned error can't find the container with id f05ab77faec5f9f9b4589ef259f2decf58cd4ac0b67d4da4549fbf2f6db18faa
Jul 19 08:21:20.433716 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:20.433614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" event={"ID":"1fa19f2c-de36-4cc9-b208-13c6866f7701","Type":"ContainerStarted","Data":"f05ab77faec5f9f9b4589ef259f2decf58cd4ac0b67d4da4549fbf2f6db18faa"}
Jul 19 08:21:21.440343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:21.440310    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerID="9c8ebc211f542c4f95b4dc88c47840266323cd428e5c1d39a29c08282e8a624a" exitCode=7
Jul 19 08:21:21.440784 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:21.440395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" event={"ID":"1fa19f2c-de36-4cc9-b208-13c6866f7701","Type":"ContainerDied","Data":"9c8ebc211f542c4f95b4dc88c47840266323cd428e5c1d39a29c08282e8a624a"}
Jul 19 08:21:21.440784 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:21.440554    2559 scope.go:117] "RemoveContainer" containerID="9c8ebc211f542c4f95b4dc88c47840266323cd428e5c1d39a29c08282e8a624a"
Jul 19 08:21:22.447490 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:22.447407    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerID="69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14" exitCode=7
Jul 19 08:21:22.447912 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:22.447489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" event={"ID":"1fa19f2c-de36-4cc9-b208-13c6866f7701","Type":"ContainerDied","Data":"69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14"}
Jul 19 08:21:22.447912 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:22.447546    2559 scope.go:117] "RemoveContainer" containerID="9c8ebc211f542c4f95b4dc88c47840266323cd428e5c1d39a29c08282e8a624a"
Jul 19 08:21:22.447912 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:22.447735    2559 scope.go:117] "RemoveContainer" containerID="69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14"
Jul 19 08:21:22.448027 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:21:22.447956    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb_odh-ai-gateway-infra(1fa19f2c-de36-4cc9-b208-13c6866f7701)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701"
Jul 19 08:21:23.454277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:23.454253    2559 scope.go:117] "RemoveContainer" containerID="69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14"
Jul 19 08:21:23.454674 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:21:23.454466    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb_odh-ai-gateway-infra(1fa19f2c-de36-4cc9-b208-13c6866f7701)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701"
Jul 19 08:21:37.455195 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:37.455121    2559 scope.go:117] "RemoveContainer" containerID="69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14"
Jul 19 08:21:38.533298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:38.533210    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerID="678b9f056903bc86e5712c5f3e13baa90c801d04a1dcaaf41fe8af759b131b11" exitCode=7
Jul 19 08:21:38.533298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:38.533272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" event={"ID":"1fa19f2c-de36-4cc9-b208-13c6866f7701","Type":"ContainerDied","Data":"678b9f056903bc86e5712c5f3e13baa90c801d04a1dcaaf41fe8af759b131b11"}
Jul 19 08:21:38.533808 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:38.533318    2559 scope.go:117] "RemoveContainer" containerID="69aba853ead42ac7e730e257fbaff1d62dbd4bfb74cbe37df23b250e31800d14"
Jul 19 08:21:38.533808 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:38.533587    2559 scope.go:117] "RemoveContainer" containerID="678b9f056903bc86e5712c5f3e13baa90c801d04a1dcaaf41fe8af759b131b11"
Jul 19 08:21:38.533912 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:21:38.533826    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb_odh-ai-gateway-infra(1fa19f2c-de36-4cc9-b208-13c6866f7701)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701"
Jul 19 08:21:39.556608 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:39.556566    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:39.690524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:39.690496    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:39.734022 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:39.733996    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca\") pod \"1fa19f2c-de36-4cc9-b208-13c6866f7701\" (UID: \"1fa19f2c-de36-4cc9-b208-13c6866f7701\") "
Jul 19 08:21:39.734318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:39.734295    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fa19f2c-de36-4cc9-b208-13c6866f7701" (UID: "1fa19f2c-de36-4cc9-b208-13c6866f7701"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:21:39.835291 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:39.835261    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fa19f2c-de36-4cc9-b208-13c6866f7701-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:21:40.546860 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:40.546835    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"
Jul 19 08:21:40.547048 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:40.546833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb" event={"ID":"1fa19f2c-de36-4cc9-b208-13c6866f7701","Type":"ContainerDied","Data":"f05ab77faec5f9f9b4589ef259f2decf58cd4ac0b67d4da4549fbf2f6db18faa"}
Jul 19 08:21:40.547048 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:40.546961    2559 scope.go:117] "RemoveContainer" containerID="678b9f056903bc86e5712c5f3e13baa90c801d04a1dcaaf41fe8af759b131b11"
Jul 19 08:21:40.562921 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:40.562896    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:40.572621 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:40.572596    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-lkjqb"]
Jul 19 08:21:42.461028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:21:42.460997    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" path="/var/lib/kubelet/pods/1fa19f2c-de36-4cc9-b208-13c6866f7701/volumes"
Jul 19 08:22:19.873439 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873403    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873807    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873822    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873841    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873847    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873860    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873866    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873871    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873941    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873949    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873955    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fa19f2c-de36-4cc9-b208-13c6866f7701" containerName="revoke-keys"
Jul 19 08:22:19.873960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.873962    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eec5204-796a-4ad6-a21d-7b10b72715cf" containerName="revoke-keys"
Jul 19 08:22:19.876880 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.876864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:19.879329 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.879302    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:22:19.879482 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.879307    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:22:19.883847 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.883826    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:19.896638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.896605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj\" (UID: \"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:19.997292 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.997259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj\" (UID: \"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:19.997947 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:19.997920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj\" (UID: \"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:20.189909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:20.189810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:20.317819 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:20.317790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:20.319339 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:22:20.319304    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd3384b_b9ae_4cb7_a0e5_00c0e5b744b4.slice/crio-01946dd1f5cf3a9910cc5bf1989406430ed65d20e94078e32c6f6423884043ca WatchSource:0}: Error finding container 01946dd1f5cf3a9910cc5bf1989406430ed65d20e94078e32c6f6423884043ca: Status 404 returned error can't find the container with id 01946dd1f5cf3a9910cc5bf1989406430ed65d20e94078e32c6f6423884043ca
Jul 19 08:22:20.763195 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:20.763090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerStarted","Data":"074c4f48fd1313951a346c0e682cb44cfa6dcfc62f2e27e09ebf40ec6e243b7c"}
Jul 19 08:22:20.763195 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:20.763144    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerStarted","Data":"01946dd1f5cf3a9910cc5bf1989406430ed65d20e94078e32c6f6423884043ca"}
Jul 19 08:22:20.778973 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:20.778926    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" podStartSLOduration=1.778912574 podStartE2EDuration="1.778912574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:22:20.776356768 +0000 UTC m=+2377.285954661" watchObservedRunningTime="2026-07-19 08:22:20.778912574 +0000 UTC m=+2377.288510473"
Jul 19 08:22:21.769533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:21.769497    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerID="074c4f48fd1313951a346c0e682cb44cfa6dcfc62f2e27e09ebf40ec6e243b7c" exitCode=7
Jul 19 08:22:21.769940 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:21.769548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerDied","Data":"074c4f48fd1313951a346c0e682cb44cfa6dcfc62f2e27e09ebf40ec6e243b7c"}
Jul 19 08:22:21.769940 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:21.769773    2559 scope.go:117] "RemoveContainer" containerID="074c4f48fd1313951a346c0e682cb44cfa6dcfc62f2e27e09ebf40ec6e243b7c"
Jul 19 08:22:22.777065 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:22.776980    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerID="ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b" exitCode=7
Jul 19 08:22:22.777454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:22.777068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerDied","Data":"ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b"}
Jul 19 08:22:22.777454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:22.777131    2559 scope.go:117] "RemoveContainer" containerID="074c4f48fd1313951a346c0e682cb44cfa6dcfc62f2e27e09ebf40ec6e243b7c"
Jul 19 08:22:22.777454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:22.777259    2559 scope.go:117] "RemoveContainer" containerID="ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b"
Jul 19 08:22:22.777587 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:22:22.777475    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj_odh-ai-gateway-infra(ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4"
Jul 19 08:22:23.783808 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:23.783783    2559 scope.go:117] "RemoveContainer" containerID="ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b"
Jul 19 08:22:23.784263 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:22:23.784030    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj_odh-ai-gateway-infra(ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4"
Jul 19 08:22:34.457649 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:34.457616    2559 scope.go:117] "RemoveContainer" containerID="ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b"
Jul 19 08:22:34.849864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:34.849820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerStarted","Data":"6693a38573ce2fd1a2c7433985397f7d24d4514ca13ed65350b88aaa9db848db"}
Jul 19 08:22:35.857218 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:35.857182    2559 generic.go:356] "Generic (PLEG): container finished" podID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerID="6693a38573ce2fd1a2c7433985397f7d24d4514ca13ed65350b88aaa9db848db" exitCode=7
Jul 19 08:22:35.857671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:35.857256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerDied","Data":"6693a38573ce2fd1a2c7433985397f7d24d4514ca13ed65350b88aaa9db848db"}
Jul 19 08:22:35.857671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:35.857368    2559 scope.go:117] "RemoveContainer" containerID="ec333373dafdabee01e91d7086b7745c837dd8cbe92be98c5a0374c09e6bd92b"
Jul 19 08:22:35.857813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:35.857767    2559 scope.go:117] "RemoveContainer" containerID="6693a38573ce2fd1a2c7433985397f7d24d4514ca13ed65350b88aaa9db848db"
Jul 19 08:22:35.857997 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:22:35.857973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj_odh-ai-gateway-infra(ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4"
Jul 19 08:22:35.873241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:35.873216    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:36.997239 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:36.997213    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:37.158343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.158248    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca\") pod \"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4\" (UID: \"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4\") "
Jul 19 08:22:37.158622 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.158599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" (UID: "ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:22:37.259556 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.259524    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:22:37.870848 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.870822    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"
Jul 19 08:22:37.871011 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.870863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj" event={"ID":"ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4","Type":"ContainerDied","Data":"01946dd1f5cf3a9910cc5bf1989406430ed65d20e94078e32c6f6423884043ca"}
Jul 19 08:22:37.871011 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.870919    2559 scope.go:117] "RemoveContainer" containerID="6693a38573ce2fd1a2c7433985397f7d24d4514ca13ed65350b88aaa9db848db"
Jul 19 08:22:37.893978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.893955    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:37.897946 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:37.897921    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-5sswj"]
Jul 19 08:22:38.460855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:22:38.460822    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" path="/var/lib/kubelet/pods/ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4/volumes"
Jul 19 08:23:13.043808 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.043727    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044067    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044077    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044105    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044110    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044163    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.044277 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.044177    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:13.047276 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.047260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:13.049738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.049660    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:23:13.050908 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.050885    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:23:13.054372 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.054351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:13.179037 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.178992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk\" (UID: \"5779427a-0e31-4644-856a-c28083256cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:13.280115 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.280079    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk\" (UID: \"5779427a-0e31-4644-856a-c28083256cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:13.280695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.280676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk\" (UID: \"5779427a-0e31-4644-856a-c28083256cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:13.359585 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.359555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:13.695560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:13.695537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:13.697624 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:23:13.697599    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5779427a_0e31_4644_856a_c28083256cd0.slice/crio-1ad53ff99025c380ef890b348bb555eea8bba9a95948fcc8d3131e0a001053d6 WatchSource:0}: Error finding container 1ad53ff99025c380ef890b348bb555eea8bba9a95948fcc8d3131e0a001053d6: Status 404 returned error can't find the container with id 1ad53ff99025c380ef890b348bb555eea8bba9a95948fcc8d3131e0a001053d6
Jul 19 08:23:14.065309 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:14.065271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerStarted","Data":"b706fcec89769acb2320946acc0c048a273501315e285a47441ff8faaa21cfd5"}
Jul 19 08:23:14.065684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:14.065327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerStarted","Data":"1ad53ff99025c380ef890b348bb555eea8bba9a95948fcc8d3131e0a001053d6"}
Jul 19 08:23:14.080353 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:14.080307    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" podStartSLOduration=1.080292048 podStartE2EDuration="1.080292048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:23:14.079391741 +0000 UTC m=+2430.588989642" watchObservedRunningTime="2026-07-19 08:23:14.080292048 +0000 UTC m=+2430.589889947"
Jul 19 08:23:15.071746 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:15.071693    2559 generic.go:356] "Generic (PLEG): container finished" podID="5779427a-0e31-4644-856a-c28083256cd0" containerID="b706fcec89769acb2320946acc0c048a273501315e285a47441ff8faaa21cfd5" exitCode=6
Jul 19 08:23:15.072190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:15.071807    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerDied","Data":"b706fcec89769acb2320946acc0c048a273501315e285a47441ff8faaa21cfd5"}
Jul 19 08:23:15.072190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:15.072070    2559 scope.go:117] "RemoveContainer" containerID="b706fcec89769acb2320946acc0c048a273501315e285a47441ff8faaa21cfd5"
Jul 19 08:23:16.078687 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:16.078652    2559 generic.go:356] "Generic (PLEG): container finished" podID="5779427a-0e31-4644-856a-c28083256cd0" containerID="25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31" exitCode=6
Jul 19 08:23:16.079089 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:16.078738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerDied","Data":"25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31"}
Jul 19 08:23:16.079089 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:16.078803    2559 scope.go:117] "RemoveContainer" containerID="b706fcec89769acb2320946acc0c048a273501315e285a47441ff8faaa21cfd5"
Jul 19 08:23:16.079089 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:16.078998    2559 scope.go:117] "RemoveContainer" containerID="25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31"
Jul 19 08:23:16.079261 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:16.079239    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk_odh-ai-gateway-infra(5779427a-0e31-4644-856a-c28083256cd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" podUID="5779427a-0e31-4644-856a-c28083256cd0"
Jul 19 08:23:17.085627 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:17.085602    2559 scope.go:117] "RemoveContainer" containerID="25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31"
Jul 19 08:23:17.086009 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:17.085809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk_odh-ai-gateway-infra(5779427a-0e31-4644-856a-c28083256cd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" podUID="5779427a-0e31-4644-856a-c28083256cd0"
Jul 19 08:23:30.454771 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:30.454738    2559 scope.go:117] "RemoveContainer" containerID="25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31"
Jul 19 08:23:31.160593 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:31.160559    2559 generic.go:356] "Generic (PLEG): container finished" podID="5779427a-0e31-4644-856a-c28083256cd0" containerID="0674611c5288871937764a49e3015c525e6d26f0eff7a8132fc11fcfd0d43702" exitCode=6
Jul 19 08:23:31.160764 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:31.160618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerDied","Data":"0674611c5288871937764a49e3015c525e6d26f0eff7a8132fc11fcfd0d43702"}
Jul 19 08:23:31.160764 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:31.160665    2559 scope.go:117] "RemoveContainer" containerID="25d4bb8892fbd3c77a64738e454fc1e822f9c727b964f6d67a0effe50148cd31"
Jul 19 08:23:31.160974 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:31.160951    2559 scope.go:117] "RemoveContainer" containerID="0674611c5288871937764a49e3015c525e6d26f0eff7a8132fc11fcfd0d43702"
Jul 19 08:23:31.161212 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:31.161188    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk_odh-ai-gateway-infra(5779427a-0e31-4644-856a-c28083256cd0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" podUID="5779427a-0e31-4644-856a-c28083256cd0"
Jul 19 08:23:32.187598 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:32.187561    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:32.322253 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:32.322231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:32.459650 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:32.459573    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca\") pod \"5779427a-0e31-4644-856a-c28083256cd0\" (UID: \"5779427a-0e31-4644-856a-c28083256cd0\") "
Jul 19 08:23:32.459951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:32.459929    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5779427a-0e31-4644-856a-c28083256cd0" (UID: "5779427a-0e31-4644-856a-c28083256cd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:23:32.560965 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:32.560935    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5779427a-0e31-4644-856a-c28083256cd0-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:23:33.176264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:33.176235    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"
Jul 19 08:23:33.176264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:33.176256    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk" event={"ID":"5779427a-0e31-4644-856a-c28083256cd0","Type":"ContainerDied","Data":"1ad53ff99025c380ef890b348bb555eea8bba9a95948fcc8d3131e0a001053d6"}
Jul 19 08:23:33.176534 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:33.176312    2559 scope.go:117] "RemoveContainer" containerID="0674611c5288871937764a49e3015c525e6d26f0eff7a8132fc11fcfd0d43702"
Jul 19 08:23:33.194652 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:33.194625    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:33.196879 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:33.196860    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-54mtk"]
Jul 19 08:23:34.461106 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.461073    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5779427a-0e31-4644-856a-c28083256cd0" path="/var/lib/kubelet/pods/5779427a-0e31-4644-856a-c28083256cd0/volumes"
Jul 19 08:23:34.875585 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.875545    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:34.876316 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876293    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876318    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876334    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876342    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876379    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:34.876436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876386    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:34.876675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876563    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876577    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:34.876675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.876594    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddd3384b-b9ae-4cb7-a0e5-00c0e5b744b4" containerName="revoke-keys"
Jul 19 08:23:34.881846 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.881820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:34.884286 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.884263    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:23:34.884483 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.884317    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:23:34.884966 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.884947    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:34.983799 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:34.983768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4\" (UID: \"d0a339ed-a3d3-4fb9-bc78-18997c6106b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:35.084246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:35.084216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4\" (UID: \"d0a339ed-a3d3-4fb9-bc78-18997c6106b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:35.084855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:35.084838    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4\" (UID: \"d0a339ed-a3d3-4fb9-bc78-18997c6106b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:35.192894 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:35.192812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:35.338253 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:35.338226    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:35.340357 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:23:35.340329    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a339ed_a3d3_4fb9_bc78_18997c6106b4.slice/crio-5633f6fe861a7c0c6277081c0fa0a48b4ed6dd3d0704922752373a9bc9a2647a WatchSource:0}: Error finding container 5633f6fe861a7c0c6277081c0fa0a48b4ed6dd3d0704922752373a9bc9a2647a: Status 404 returned error can't find the container with id 5633f6fe861a7c0c6277081c0fa0a48b4ed6dd3d0704922752373a9bc9a2647a
Jul 19 08:23:36.194210 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:36.194176    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerID="cfb77e50d5982fc1d00c4ed568a51e5af48366514fe495465f531efa0717f71b" exitCode=6
Jul 19 08:23:36.194592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:36.194236    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" event={"ID":"d0a339ed-a3d3-4fb9-bc78-18997c6106b4","Type":"ContainerDied","Data":"cfb77e50d5982fc1d00c4ed568a51e5af48366514fe495465f531efa0717f71b"}
Jul 19 08:23:36.194592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:36.194295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" event={"ID":"d0a339ed-a3d3-4fb9-bc78-18997c6106b4","Type":"ContainerStarted","Data":"5633f6fe861a7c0c6277081c0fa0a48b4ed6dd3d0704922752373a9bc9a2647a"}
Jul 19 08:23:36.194592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:36.194393    2559 scope.go:117] "RemoveContainer" containerID="cfb77e50d5982fc1d00c4ed568a51e5af48366514fe495465f531efa0717f71b"
Jul 19 08:23:37.201436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:37.201400    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerID="a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d" exitCode=6
Jul 19 08:23:37.201436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:37.201440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" event={"ID":"d0a339ed-a3d3-4fb9-bc78-18997c6106b4","Type":"ContainerDied","Data":"a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d"}
Jul 19 08:23:37.201932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:37.201498    2559 scope.go:117] "RemoveContainer" containerID="cfb77e50d5982fc1d00c4ed568a51e5af48366514fe495465f531efa0717f71b"
Jul 19 08:23:37.201932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:37.201654    2559 scope.go:117] "RemoveContainer" containerID="a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d"
Jul 19 08:23:37.201932 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:37.201916    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4_odh-ai-gateway-infra(d0a339ed-a3d3-4fb9-bc78-18997c6106b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4"
Jul 19 08:23:38.208726 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:38.208673    2559 scope.go:117] "RemoveContainer" containerID="a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d"
Jul 19 08:23:38.209117 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:38.208933    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4_odh-ai-gateway-infra(d0a339ed-a3d3-4fb9-bc78-18997c6106b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4"
Jul 19 08:23:53.455416 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:53.455384    2559 scope.go:117] "RemoveContainer" containerID="a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d"
Jul 19 08:23:54.298797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:54.298766    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerID="aaddad296842d0df558f9a9b7ed8e8041bc94018a3f93bdded068e4e3c74297c" exitCode=6
Jul 19 08:23:54.298968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:54.298849    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" event={"ID":"d0a339ed-a3d3-4fb9-bc78-18997c6106b4","Type":"ContainerDied","Data":"aaddad296842d0df558f9a9b7ed8e8041bc94018a3f93bdded068e4e3c74297c"}
Jul 19 08:23:54.298968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:54.298934    2559 scope.go:117] "RemoveContainer" containerID="a641a35394a128d7b2acfcc0905bc4ccc1234a5bdc40b93bccccb4147865122d"
Jul 19 08:23:54.299254 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:54.299230    2559 scope.go:117] "RemoveContainer" containerID="aaddad296842d0df558f9a9b7ed8e8041bc94018a3f93bdded068e4e3c74297c"
Jul 19 08:23:54.299468 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:23:54.299446    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4_odh-ai-gateway-infra(d0a339ed-a3d3-4fb9-bc78-18997c6106b4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4"
Jul 19 08:23:55.322472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:55.322440    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:55.457712 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:55.457678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:55.565863 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:55.565833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca\") pod \"d0a339ed-a3d3-4fb9-bc78-18997c6106b4\" (UID: \"d0a339ed-a3d3-4fb9-bc78-18997c6106b4\") "
Jul 19 08:23:55.566157 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:55.566135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d0a339ed-a3d3-4fb9-bc78-18997c6106b4" (UID: "d0a339ed-a3d3-4fb9-bc78-18997c6106b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:23:55.666999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:55.666924    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a339ed-a3d3-4fb9-bc78-18997c6106b4-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:23:56.312594 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.312565    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"
Jul 19 08:23:56.312819 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.312564    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4" event={"ID":"d0a339ed-a3d3-4fb9-bc78-18997c6106b4","Type":"ContainerDied","Data":"5633f6fe861a7c0c6277081c0fa0a48b4ed6dd3d0704922752373a9bc9a2647a"}
Jul 19 08:23:56.312819 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.312722    2559 scope.go:117] "RemoveContainer" containerID="aaddad296842d0df558f9a9b7ed8e8041bc94018a3f93bdded068e4e3c74297c"
Jul 19 08:23:56.334673 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.334645    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:56.338381 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.338358    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-g5zf4"]
Jul 19 08:23:56.461043 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:56.461013    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" path="/var/lib/kubelet/pods/d0a339ed-a3d3-4fb9-bc78-18997c6106b4/volumes"
Jul 19 08:23:59.958224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958185    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:23:59.958692 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958676    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.958692 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958694    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958728    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958738    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958751    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958759    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958778    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.958864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958786    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.959149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958899    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.959149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958915    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:23:59.959149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.958926    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5779427a-0e31-4644-856a-c28083256cd0" containerName="revoke-keys"
Jul 19 08:23:59.963579 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.963558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:23:59.966269 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.966241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:23:59.966398 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.966294    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:23:59.967849 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:23:59.967828    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:24:00.105852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:00.105817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv\" (UID: \"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:00.206956 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:00.206921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv\" (UID: \"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:00.207605 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:00.207584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv\" (UID: \"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:00.275348 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:00.275257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:00.407601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:00.407578    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:24:00.409332 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:24:00.409302    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d5e124_5794_4b3b_b42c_fbe09b1bcf6a.slice/crio-1b0899d27127e5d53130b24b28907baf5c7532d775dc5331ec8c56e74d972f8f WatchSource:0}: Error finding container 1b0899d27127e5d53130b24b28907baf5c7532d775dc5331ec8c56e74d972f8f: Status 404 returned error can't find the container with id 1b0899d27127e5d53130b24b28907baf5c7532d775dc5331ec8c56e74d972f8f
Jul 19 08:24:01.343861 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:01.343825    2559 generic.go:356] "Generic (PLEG): container finished" podID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerID="1dd59a2b12ac80a6d6a782f410e9112093653984794ffc6cdd9a3fda1d2efaf5" exitCode=7
Jul 19 08:24:01.344278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:01.343911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" event={"ID":"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a","Type":"ContainerDied","Data":"1dd59a2b12ac80a6d6a782f410e9112093653984794ffc6cdd9a3fda1d2efaf5"}
Jul 19 08:24:01.344278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:01.343969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" event={"ID":"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a","Type":"ContainerStarted","Data":"1b0899d27127e5d53130b24b28907baf5c7532d775dc5331ec8c56e74d972f8f"}
Jul 19 08:24:01.344278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:01.344114    2559 scope.go:117] "RemoveContainer" containerID="1dd59a2b12ac80a6d6a782f410e9112093653984794ffc6cdd9a3fda1d2efaf5"
Jul 19 08:24:02.351174 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:02.351141    2559 generic.go:356] "Generic (PLEG): container finished" podID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerID="b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b" exitCode=7
Jul 19 08:24:02.351670 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:02.351204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" event={"ID":"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a","Type":"ContainerDied","Data":"b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b"}
Jul 19 08:24:02.351670 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:02.351252    2559 scope.go:117] "RemoveContainer" containerID="1dd59a2b12ac80a6d6a782f410e9112093653984794ffc6cdd9a3fda1d2efaf5"
Jul 19 08:24:02.351670 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:02.351451    2559 scope.go:117] "RemoveContainer" containerID="b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b"
Jul 19 08:24:02.351670 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:02.351662    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv_odh-ai-gateway-infra(50d5e124-5794-4b3b-b42c-fbe09b1bcf6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a"
Jul 19 08:24:03.357488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:03.357462    2559 scope.go:117] "RemoveContainer" containerID="b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b"
Jul 19 08:24:03.357903 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:03.357638    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv_odh-ai-gateway-infra(50d5e124-5794-4b3b-b42c-fbe09b1bcf6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a"
Jul 19 08:24:15.455555 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:15.455524    2559 scope.go:117] "RemoveContainer" containerID="b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b"
Jul 19 08:24:16.429610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:16.429523    2559 generic.go:356] "Generic (PLEG): container finished" podID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerID="6a6afdb5e72b60a7031eea5c6425e7a4616393e5762feddc3575c671204c58b3" exitCode=7
Jul 19 08:24:16.429767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:16.429607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" event={"ID":"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a","Type":"ContainerDied","Data":"6a6afdb5e72b60a7031eea5c6425e7a4616393e5762feddc3575c671204c58b3"}
Jul 19 08:24:16.429767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:16.429674    2559 scope.go:117] "RemoveContainer" containerID="b96bd3b3d00180af44d5bff09c61c75576be48d07ab85dfbab194acae8a0cd7b"
Jul 19 08:24:16.429982 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:16.429967    2559 scope.go:117] "RemoveContainer" containerID="6a6afdb5e72b60a7031eea5c6425e7a4616393e5762feddc3575c671204c58b3"
Jul 19 08:24:16.430218 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:16.430199    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv_odh-ai-gateway-infra(50d5e124-5794-4b3b-b42c-fbe09b1bcf6a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a"
Jul 19 08:24:17.455080 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:17.455050    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:24:17.584536 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:17.584513    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:17.768278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:17.768196    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca\") pod \"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a\" (UID: \"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a\") "
Jul 19 08:24:17.768525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:17.768501    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" (UID: "50d5e124-5794-4b3b-b42c-fbe09b1bcf6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:24:17.869999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:17.869969    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:24:18.449146 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:18.449109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv" event={"ID":"50d5e124-5794-4b3b-b42c-fbe09b1bcf6a","Type":"ContainerDied","Data":"1b0899d27127e5d53130b24b28907baf5c7532d775dc5331ec8c56e74d972f8f"}
Jul 19 08:24:18.449337 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:18.449193    2559 scope.go:117] "RemoveContainer" containerID="6a6afdb5e72b60a7031eea5c6425e7a4616393e5762feddc3575c671204c58b3"
Jul 19 08:24:18.449337 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:18.449121    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"
Jul 19 08:24:18.472217 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:18.472186    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:24:18.477115 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:18.477091    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-nc7vv"]
Jul 19 08:24:20.461000 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:20.460969    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" path="/var/lib/kubelet/pods/50d5e124-5794-4b3b-b42c-fbe09b1bcf6a/volumes"
Jul 19 08:24:34.924201 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924164    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:34.924810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924727    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.924810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924746    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.924810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924757    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.924810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924766    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.925046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924856    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.925046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924870    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0a339ed-a3d3-4fb9-bc78-18997c6106b4" containerName="revoke-keys"
Jul 19 08:24:34.925046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924881    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.925046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.924891    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:24:34.928087 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.928068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:34.930420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.930400    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:24:34.930746 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.930732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:24:34.935155 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:34.935132    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:35.017018 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.016986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6\" (UID: \"767e303e-862f-42ff-b9a6-bcb9b3b0a932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:35.118024 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.117996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6\" (UID: \"767e303e-862f-42ff-b9a6-bcb9b3b0a932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:35.118610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.118585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6\" (UID: \"767e303e-862f-42ff-b9a6-bcb9b3b0a932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:35.239985 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.239901    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:35.376526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.376487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:35.379509 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:24:35.379480    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767e303e_862f_42ff_b9a6_bcb9b3b0a932.slice/crio-fdb3be3b4666b770710fb5fa14671ae16a155bcde8c68fe01c9f8a7042aa7b24 WatchSource:0}: Error finding container fdb3be3b4666b770710fb5fa14671ae16a155bcde8c68fe01c9f8a7042aa7b24: Status 404 returned error can't find the container with id fdb3be3b4666b770710fb5fa14671ae16a155bcde8c68fe01c9f8a7042aa7b24
Jul 19 08:24:35.540824 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:35.540738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" event={"ID":"767e303e-862f-42ff-b9a6-bcb9b3b0a932","Type":"ContainerStarted","Data":"fdb3be3b4666b770710fb5fa14671ae16a155bcde8c68fe01c9f8a7042aa7b24"}
Jul 19 08:24:36.547447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:36.547409    2559 generic.go:356] "Generic (PLEG): container finished" podID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerID="f315b3d6aebf23056db3f2ce93d7ee242e8d2a8d0b8064750ee7e8d8a02774e7" exitCode=6
Jul 19 08:24:36.547854 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:36.547500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" event={"ID":"767e303e-862f-42ff-b9a6-bcb9b3b0a932","Type":"ContainerDied","Data":"f315b3d6aebf23056db3f2ce93d7ee242e8d2a8d0b8064750ee7e8d8a02774e7"}
Jul 19 08:24:36.547854 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:36.547673    2559 scope.go:117] "RemoveContainer" containerID="f315b3d6aebf23056db3f2ce93d7ee242e8d2a8d0b8064750ee7e8d8a02774e7"
Jul 19 08:24:37.555238 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:37.555142    2559 generic.go:356] "Generic (PLEG): container finished" podID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerID="454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6" exitCode=6
Jul 19 08:24:37.555238 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:37.555227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" event={"ID":"767e303e-862f-42ff-b9a6-bcb9b3b0a932","Type":"ContainerDied","Data":"454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6"}
Jul 19 08:24:37.555749 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:37.555282    2559 scope.go:117] "RemoveContainer" containerID="f315b3d6aebf23056db3f2ce93d7ee242e8d2a8d0b8064750ee7e8d8a02774e7"
Jul 19 08:24:37.555749 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:37.555480    2559 scope.go:117] "RemoveContainer" containerID="454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6"
Jul 19 08:24:37.555749 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:37.555685    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6_odh-ai-gateway-infra(767e303e-862f-42ff-b9a6-bcb9b3b0a932)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932"
Jul 19 08:24:38.562241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:38.562213    2559 scope.go:117] "RemoveContainer" containerID="454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6"
Jul 19 08:24:38.562618 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:38.562387    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6_odh-ai-gateway-infra(767e303e-862f-42ff-b9a6-bcb9b3b0a932)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932"
Jul 19 08:24:49.455025 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:49.454989    2559 scope.go:117] "RemoveContainer" containerID="454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6"
Jul 19 08:24:50.626869 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:50.626833    2559 generic.go:356] "Generic (PLEG): container finished" podID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerID="b0c53da5ff5ac4db1e6d3c78a51435b182110a65f68ba06892fb44b937ae3554" exitCode=6
Jul 19 08:24:50.627250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:50.626906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" event={"ID":"767e303e-862f-42ff-b9a6-bcb9b3b0a932","Type":"ContainerDied","Data":"b0c53da5ff5ac4db1e6d3c78a51435b182110a65f68ba06892fb44b937ae3554"}
Jul 19 08:24:50.627250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:50.626970    2559 scope.go:117] "RemoveContainer" containerID="454c016d01f614f883417053911ecf4e6476c228b97766763bb153d0c775f0b6"
Jul 19 08:24:50.627406 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:50.627375    2559 scope.go:117] "RemoveContainer" containerID="b0c53da5ff5ac4db1e6d3c78a51435b182110a65f68ba06892fb44b937ae3554"
Jul 19 08:24:50.627631 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:24:50.627601    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6_odh-ai-gateway-infra(767e303e-862f-42ff-b9a6-bcb9b3b0a932)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932"
Jul 19 08:24:51.651311 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:51.651275    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:51.789944 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:51.789920    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:51.876462 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:51.876429    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca\") pod \"767e303e-862f-42ff-b9a6-bcb9b3b0a932\" (UID: \"767e303e-862f-42ff-b9a6-bcb9b3b0a932\") "
Jul 19 08:24:51.876837 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:51.876818    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "767e303e-862f-42ff-b9a6-bcb9b3b0a932" (UID: "767e303e-862f-42ff-b9a6-bcb9b3b0a932"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:24:51.978063 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:51.977981    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767e303e-862f-42ff-b9a6-bcb9b3b0a932-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:24:52.640263 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:52.640237    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"
Jul 19 08:24:52.640431 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:52.640230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6" event={"ID":"767e303e-862f-42ff-b9a6-bcb9b3b0a932","Type":"ContainerDied","Data":"fdb3be3b4666b770710fb5fa14671ae16a155bcde8c68fe01c9f8a7042aa7b24"}
Jul 19 08:24:52.640431 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:52.640374    2559 scope.go:117] "RemoveContainer" containerID="b0c53da5ff5ac4db1e6d3c78a51435b182110a65f68ba06892fb44b937ae3554"
Jul 19 08:24:52.659653 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:52.659614    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:52.665019 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:52.664991    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-8s7q6"]
Jul 19 08:24:54.462231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:24:54.462198    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" path="/var/lib/kubelet/pods/767e303e-862f-42ff-b9a6-bcb9b3b0a932/volumes"
Jul 19 08:25:03.130419 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130386    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130740    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130753    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50d5e124-5794-4b3b-b42c-fbe09b1bcf6a" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130770    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130775    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130784    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130790    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130798    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.130813 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130803    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.131069 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130858    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.131069 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.130866    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:03.135065 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.135047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:03.138077 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.138052    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:25:03.139272 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.139250    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:25:03.142359 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.142331    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:03.276735 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.276687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt\" (UID: \"f87b18e7-9df5-4bb4-9665-407a51d87411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:03.378223 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.378183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt\" (UID: \"f87b18e7-9df5-4bb4-9665-407a51d87411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:03.378822 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.378804    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt\" (UID: \"f87b18e7-9df5-4bb4-9665-407a51d87411\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:03.448430 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.448340    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:03.576305 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.576281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:03.578317 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:25:03.578293    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87b18e7_9df5_4bb4_9665_407a51d87411.slice/crio-47a8384b6184af4bea294276987e7d589fb8a76e831886ce8a45568099d61347 WatchSource:0}: Error finding container 47a8384b6184af4bea294276987e7d589fb8a76e831886ce8a45568099d61347: Status 404 returned error can't find the container with id 47a8384b6184af4bea294276987e7d589fb8a76e831886ce8a45568099d61347
Jul 19 08:25:03.698851 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:03.698767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerStarted","Data":"47a8384b6184af4bea294276987e7d589fb8a76e831886ce8a45568099d61347"}
Jul 19 08:25:04.706587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:04.706555    2559 generic.go:356] "Generic (PLEG): container finished" podID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerID="98121c8654f3b95e566a633cb397956a597bc9c8be787c40eec5b49a261a5394" exitCode=6
Jul 19 08:25:04.706995 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:04.706604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerDied","Data":"98121c8654f3b95e566a633cb397956a597bc9c8be787c40eec5b49a261a5394"}
Jul 19 08:25:04.706995 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:04.706868    2559 scope.go:117] "RemoveContainer" containerID="98121c8654f3b95e566a633cb397956a597bc9c8be787c40eec5b49a261a5394"
Jul 19 08:25:05.714413 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:05.714328    2559 generic.go:356] "Generic (PLEG): container finished" podID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerID="537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa" exitCode=6
Jul 19 08:25:05.714817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:05.714414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerDied","Data":"537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa"}
Jul 19 08:25:05.714817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:05.714470    2559 scope.go:117] "RemoveContainer" containerID="98121c8654f3b95e566a633cb397956a597bc9c8be787c40eec5b49a261a5394"
Jul 19 08:25:05.714817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:05.714662    2559 scope.go:117] "RemoveContainer" containerID="537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa"
Jul 19 08:25:05.714936 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:05.714892    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt_odh-ai-gateway-infra(f87b18e7-9df5-4bb4-9665-407a51d87411)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411"
Jul 19 08:25:06.721917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:06.721889    2559 scope.go:117] "RemoveContainer" containerID="537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa"
Jul 19 08:25:06.722394 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:06.722054    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt_odh-ai-gateway-infra(f87b18e7-9df5-4bb4-9665-407a51d87411)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411"
Jul 19 08:25:20.455068 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:20.455032    2559 scope.go:117] "RemoveContainer" containerID="537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa"
Jul 19 08:25:20.796014 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:20.795984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerStarted","Data":"d2fd50121afd91d7555d1920b5c08a8bc69e77230ad269be602a7f3d5990cb8d"}
Jul 19 08:25:20.810830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:20.810783    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" podStartSLOduration=17.810767693 podStartE2EDuration="17.810767693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:25:20.809532063 +0000 UTC m=+2557.319129964" watchObservedRunningTime="2026-07-19 08:25:20.810767693 +0000 UTC m=+2557.320365625"
Jul 19 08:25:21.803202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:21.803167    2559 generic.go:356] "Generic (PLEG): container finished" podID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerID="d2fd50121afd91d7555d1920b5c08a8bc69e77230ad269be602a7f3d5990cb8d" exitCode=6
Jul 19 08:25:21.803639 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:21.803243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerDied","Data":"d2fd50121afd91d7555d1920b5c08a8bc69e77230ad269be602a7f3d5990cb8d"}
Jul 19 08:25:21.803639 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:21.803293    2559 scope.go:117] "RemoveContainer" containerID="537d5e126c696502a950be76374ec390b12c77fda4a1afa5a51858f507b68eaa"
Jul 19 08:25:21.803774 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:21.803646    2559 scope.go:117] "RemoveContainer" containerID="d2fd50121afd91d7555d1920b5c08a8bc69e77230ad269be602a7f3d5990cb8d"
Jul 19 08:25:21.803913 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:21.803876    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt_odh-ai-gateway-infra(f87b18e7-9df5-4bb4-9665-407a51d87411)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411"
Jul 19 08:25:21.826571 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:21.826546    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:22.943643 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:22.943617    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:23.053976 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.053935    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca\") pod \"f87b18e7-9df5-4bb4-9665-407a51d87411\" (UID: \"f87b18e7-9df5-4bb4-9665-407a51d87411\") "
Jul 19 08:25:23.054306 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.054283    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f87b18e7-9df5-4bb4-9665-407a51d87411" (UID: "f87b18e7-9df5-4bb4-9665-407a51d87411"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:25:23.155678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.155602    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87b18e7-9df5-4bb4-9665-407a51d87411-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:25:23.817359 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.817332    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"
Jul 19 08:25:23.817562 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.817331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt" event={"ID":"f87b18e7-9df5-4bb4-9665-407a51d87411","Type":"ContainerDied","Data":"47a8384b6184af4bea294276987e7d589fb8a76e831886ce8a45568099d61347"}
Jul 19 08:25:23.817562 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.817460    2559 scope.go:117] "RemoveContainer" containerID="d2fd50121afd91d7555d1920b5c08a8bc69e77230ad269be602a7f3d5990cb8d"
Jul 19 08:25:23.838776 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.838751    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:23.842399 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:23.842375    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-fmltt"]
Jul 19 08:25:24.461149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.461119    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" path="/var/lib/kubelet/pods/f87b18e7-9df5-4bb4-9665-407a51d87411/volumes"
Jul 19 08:25:24.957656 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.957622    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:24.958000 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.957988    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.958042 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958002    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.958042 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958014    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.958042 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958019    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.958152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958089    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="767e303e-862f-42ff-b9a6-bcb9b3b0a932" containerName="revoke-keys"
Jul 19 08:25:24.958152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958097    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.958152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.958105    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:24.962340 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.962323    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:24.965033 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.965006    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:25:24.965167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.965033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:25:24.971193 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:24.971166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:25.074044 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.074009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8\" (UID: \"d4f89834-a9d7-4733-ac15-82e131e4e9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:25.175441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.175376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8\" (UID: \"d4f89834-a9d7-4733-ac15-82e131e4e9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:25.175999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.175979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8\" (UID: \"d4f89834-a9d7-4733-ac15-82e131e4e9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:25.274621 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.274536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:25.402603 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.402577    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:25.404281 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:25:25.404256    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f89834_a9d7_4733_ac15_82e131e4e9f4.slice/crio-3dddc84fbc9d6cb5a58fc04d8310123c35ceeb51c0d4deab529709385d789daf WatchSource:0}: Error finding container 3dddc84fbc9d6cb5a58fc04d8310123c35ceeb51c0d4deab529709385d789daf: Status 404 returned error can't find the container with id 3dddc84fbc9d6cb5a58fc04d8310123c35ceeb51c0d4deab529709385d789daf
Jul 19 08:25:25.830436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.830353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerStarted","Data":"fceab4e7b57e6fe48229f56dfc43b20207f13d6b019d8418883d4e05b6ce421f"}
Jul 19 08:25:25.830436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.830404    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerStarted","Data":"3dddc84fbc9d6cb5a58fc04d8310123c35ceeb51c0d4deab529709385d789daf"}
Jul 19 08:25:25.846474 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:25.846402    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" podStartSLOduration=1.8463854180000001 podStartE2EDuration="1.846385418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:25:25.845674811 +0000 UTC m=+2562.355272712" watchObservedRunningTime="2026-07-19 08:25:25.846385418 +0000 UTC m=+2562.355983319"
Jul 19 08:25:26.837874 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:26.837844    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerID="fceab4e7b57e6fe48229f56dfc43b20207f13d6b019d8418883d4e05b6ce421f" exitCode=6
Jul 19 08:25:26.838258 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:26.837890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerDied","Data":"fceab4e7b57e6fe48229f56dfc43b20207f13d6b019d8418883d4e05b6ce421f"}
Jul 19 08:25:26.838258 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:26.838084    2559 scope.go:117] "RemoveContainer" containerID="fceab4e7b57e6fe48229f56dfc43b20207f13d6b019d8418883d4e05b6ce421f"
Jul 19 08:25:27.845464 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:27.845432    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerID="d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932" exitCode=6
Jul 19 08:25:27.845881 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:27.845519    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerDied","Data":"d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932"}
Jul 19 08:25:27.845881 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:27.845579    2559 scope.go:117] "RemoveContainer" containerID="fceab4e7b57e6fe48229f56dfc43b20207f13d6b019d8418883d4e05b6ce421f"
Jul 19 08:25:27.845881 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:27.845742    2559 scope.go:117] "RemoveContainer" containerID="d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932"
Jul 19 08:25:27.846005 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:27.845972    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8_odh-ai-gateway-infra(d4f89834-a9d7-4733-ac15-82e131e4e9f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4"
Jul 19 08:25:28.852772 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:28.852743    2559 scope.go:117] "RemoveContainer" containerID="d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932"
Jul 19 08:25:28.853174 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:28.852929    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8_odh-ai-gateway-infra(d4f89834-a9d7-4733-ac15-82e131e4e9f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4"
Jul 19 08:25:40.454970 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:40.454938    2559 scope.go:117] "RemoveContainer" containerID="d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932"
Jul 19 08:25:40.922347 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:40.922311    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerStarted","Data":"b6bf423830c20c313cfd47663af6fa295d000b110f00af3d499e69b559d95714"}
Jul 19 08:25:41.929718 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:41.929671    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerID="b6bf423830c20c313cfd47663af6fa295d000b110f00af3d499e69b559d95714" exitCode=6
Jul 19 08:25:41.930150 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:41.929739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerDied","Data":"b6bf423830c20c313cfd47663af6fa295d000b110f00af3d499e69b559d95714"}
Jul 19 08:25:41.930150 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:41.929798    2559 scope.go:117] "RemoveContainer" containerID="d078d55cfa82e109c235923334eabde297edb1d616c138ac47f2901324f37932"
Jul 19 08:25:41.930150 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:41.930063    2559 scope.go:117] "RemoveContainer" containerID="b6bf423830c20c313cfd47663af6fa295d000b110f00af3d499e69b559d95714"
Jul 19 08:25:41.930280 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:41.930262    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8_odh-ai-gateway-infra(d4f89834-a9d7-4733-ac15-82e131e4e9f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4"
Jul 19 08:25:41.950939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:41.950916    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:43.070788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.070762    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:43.141921 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.141891    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca\") pod \"d4f89834-a9d7-4733-ac15-82e131e4e9f4\" (UID: \"d4f89834-a9d7-4733-ac15-82e131e4e9f4\") "
Jul 19 08:25:43.142262 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.142243    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4f89834-a9d7-4733-ac15-82e131e4e9f4" (UID: "d4f89834-a9d7-4733-ac15-82e131e4e9f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:25:43.243303 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.243225    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f89834-a9d7-4733-ac15-82e131e4e9f4-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:25:43.944419 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.944394    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"
Jul 19 08:25:43.944589 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.944388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8" event={"ID":"d4f89834-a9d7-4733-ac15-82e131e4e9f4","Type":"ContainerDied","Data":"3dddc84fbc9d6cb5a58fc04d8310123c35ceeb51c0d4deab529709385d789daf"}
Jul 19 08:25:43.944589 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.944535    2559 scope.go:117] "RemoveContainer" containerID="b6bf423830c20c313cfd47663af6fa295d000b110f00af3d499e69b559d95714"
Jul 19 08:25:43.966334 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.966306    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:43.971675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:43.971648    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-lfdf8"]
Jul 19 08:25:44.461833 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:44.461799    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" path="/var/lib/kubelet/pods/d4f89834-a9d7-4733-ac15-82e131e4e9f4/volumes"
Jul 19 08:25:50.046826 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.046797    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047164    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047176    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047187    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047205    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047210    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047218    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047223    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.047525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047290    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87b18e7-9df5-4bb4-9665-407a51d87411" containerName="revoke-keys"
Jul 19 08:25:50.047525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.047295    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:25:50.050193 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.050178    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:25:50.052616 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.052598    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:25:50.052740 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.052631    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:25:50.058347 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.058320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:25:50.206937 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.206901    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r\" (UID: \"f8b4b868-50ce-4419-9303-d5c319551229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:25:50.308060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.307967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r\" (UID: \"f8b4b868-50ce-4419-9303-d5c319551229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:25:50.308642 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.308618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r\" (UID: \"f8b4b868-50ce-4419-9303-d5c319551229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:25:50.361499 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.361464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:25:50.496155 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.496122    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:25:50.497731 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:25:50.497673    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b4b868_50ce_4419_9303_d5c319551229.slice/crio-11647a838d3502bdd68a9a88f177ecfb5c9041d18578877e21979dc075c11259 WatchSource:0}: Error finding container 11647a838d3502bdd68a9a88f177ecfb5c9041d18578877e21979dc075c11259: Status 404 returned error can't find the container with id 11647a838d3502bdd68a9a88f177ecfb5c9041d18578877e21979dc075c11259
Jul 19 08:25:50.983984 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.983891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerStarted","Data":"cfd565f2c88f7a2a2d53522b4e8933b6c79ddcee4134dfb1ee112cb0d2dbcf93"}
Jul 19 08:25:50.983984 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:50.983953    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerStarted","Data":"11647a838d3502bdd68a9a88f177ecfb5c9041d18578877e21979dc075c11259"}
Jul 19 08:25:51.002030 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:51.001981    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" podStartSLOduration=1.001965215 podStartE2EDuration="1.001965215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:25:51.0008045 +0000 UTC m=+2587.510402391" watchObservedRunningTime="2026-07-19 08:25:51.001965215 +0000 UTC m=+2587.511563133"
Jul 19 08:25:51.990764 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:51.990728    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8b4b868-50ce-4419-9303-d5c319551229" containerID="cfd565f2c88f7a2a2d53522b4e8933b6c79ddcee4134dfb1ee112cb0d2dbcf93" exitCode=6
Jul 19 08:25:51.991207 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:51.990782    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerDied","Data":"cfd565f2c88f7a2a2d53522b4e8933b6c79ddcee4134dfb1ee112cb0d2dbcf93"}
Jul 19 08:25:51.991207 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:51.991035    2559 scope.go:117] "RemoveContainer" containerID="cfd565f2c88f7a2a2d53522b4e8933b6c79ddcee4134dfb1ee112cb0d2dbcf93"
Jul 19 08:25:52.997365 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:52.997270    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8b4b868-50ce-4419-9303-d5c319551229" containerID="61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9" exitCode=6
Jul 19 08:25:52.997760 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:52.997357    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerDied","Data":"61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9"}
Jul 19 08:25:52.997760 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:52.997444    2559 scope.go:117] "RemoveContainer" containerID="cfd565f2c88f7a2a2d53522b4e8933b6c79ddcee4134dfb1ee112cb0d2dbcf93"
Jul 19 08:25:52.997760 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:52.997535    2559 scope.go:117] "RemoveContainer" containerID="61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9"
Jul 19 08:25:52.997913 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:52.997794    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r_odh-ai-gateway-infra(f8b4b868-50ce-4419-9303-d5c319551229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" podUID="f8b4b868-50ce-4419-9303-d5c319551229"
Jul 19 08:25:54.004322 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:25:54.004298    2559 scope.go:117] "RemoveContainer" containerID="61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9"
Jul 19 08:25:54.004694 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:25:54.004496    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r_odh-ai-gateway-infra(f8b4b868-50ce-4419-9303-d5c319551229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" podUID="f8b4b868-50ce-4419-9303-d5c319551229"
Jul 19 08:26:08.455285 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:08.455199    2559 scope.go:117] "RemoveContainer" containerID="61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9"
Jul 19 08:26:09.089072 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:09.089026    2559 generic.go:356] "Generic (PLEG): container finished" podID="f8b4b868-50ce-4419-9303-d5c319551229" containerID="a1db552fec4f2b5169c3f7ec2a03b9ca4aa82742a74dd67bdd817971001aef56" exitCode=6
Jul 19 08:26:09.089219 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:09.089101    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerDied","Data":"a1db552fec4f2b5169c3f7ec2a03b9ca4aa82742a74dd67bdd817971001aef56"}
Jul 19 08:26:09.089219 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:09.089183    2559 scope.go:117] "RemoveContainer" containerID="61153356d50af0ddef9fcda6cc396dc014f9b05b393d224c0f86dd218775aaa9"
Jul 19 08:26:09.089571 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:09.089528    2559 scope.go:117] "RemoveContainer" containerID="a1db552fec4f2b5169c3f7ec2a03b9ca4aa82742a74dd67bdd817971001aef56"
Jul 19 08:26:09.089827 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:09.089797    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r_odh-ai-gateway-infra(f8b4b868-50ce-4419-9303-d5c319551229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" podUID="f8b4b868-50ce-4419-9303-d5c319551229"
Jul 19 08:26:10.125188 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:10.125148    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:26:10.261456 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:10.261433    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:26:10.292463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:10.292435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca\") pod \"f8b4b868-50ce-4419-9303-d5c319551229\" (UID: \"f8b4b868-50ce-4419-9303-d5c319551229\") "
Jul 19 08:26:10.292774 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:10.292753    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8b4b868-50ce-4419-9303-d5c319551229" (UID: "f8b4b868-50ce-4419-9303-d5c319551229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:26:10.394029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:10.393944    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b4b868-50ce-4419-9303-d5c319551229-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:26:11.109594 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:11.109562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"
Jul 19 08:26:11.109786 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:11.109597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r" event={"ID":"f8b4b868-50ce-4419-9303-d5c319551229","Type":"ContainerDied","Data":"11647a838d3502bdd68a9a88f177ecfb5c9041d18578877e21979dc075c11259"}
Jul 19 08:26:11.109786 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:11.109683    2559 scope.go:117] "RemoveContainer" containerID="a1db552fec4f2b5169c3f7ec2a03b9ca4aa82742a74dd67bdd817971001aef56"
Jul 19 08:26:11.126795 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:11.126771    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:26:11.130512 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:11.130478    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tmv4r"]
Jul 19 08:26:12.460858 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:12.460827    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8b4b868-50ce-4419-9303-d5c319551229" path="/var/lib/kubelet/pods/f8b4b868-50ce-4419-9303-d5c319551229/volumes"
Jul 19 08:26:15.003465 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.003424    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:15.003964 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.003946    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.004034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.003966    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.004034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.004001    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.004034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.004013    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.004190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.004098    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.004190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.004112    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f89834-a9d7-4733-ac15-82e131e4e9f4" containerName="revoke-keys"
Jul 19 08:26:15.004190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.004126    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:15.008659 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.008639    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:15.011231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.011202    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:26:15.011556 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.011535    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:26:15.013772 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.013749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:15.135094 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.135063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-87h77\" (UID: \"cbb66123-374b-4eb0-b5b9-a5abf7ae8796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:15.236426 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.236391    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-87h77\" (UID: \"cbb66123-374b-4eb0-b5b9-a5abf7ae8796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:15.237031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.237013    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-87h77\" (UID: \"cbb66123-374b-4eb0-b5b9-a5abf7ae8796\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:15.320318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.320292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:15.449542 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:15.449517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:15.451557 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:26:15.451529    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb66123_374b_4eb0_b5b9_a5abf7ae8796.slice/crio-8d389cba927a6126f01d9714753bdce5a57cf90ebbee722bf6c95a6200a3302e WatchSource:0}: Error finding container 8d389cba927a6126f01d9714753bdce5a57cf90ebbee722bf6c95a6200a3302e: Status 404 returned error can't find the container with id 8d389cba927a6126f01d9714753bdce5a57cf90ebbee722bf6c95a6200a3302e
Jul 19 08:26:16.136607 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:16.136578    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerID="d94f116521bc13ad32ae17bea75a794ca91c0df832760f6f54a3a79f7d63c549" exitCode=6
Jul 19 08:26:16.137023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:16.136663    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" event={"ID":"cbb66123-374b-4eb0-b5b9-a5abf7ae8796","Type":"ContainerDied","Data":"d94f116521bc13ad32ae17bea75a794ca91c0df832760f6f54a3a79f7d63c549"}
Jul 19 08:26:16.137023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:16.136744    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" event={"ID":"cbb66123-374b-4eb0-b5b9-a5abf7ae8796","Type":"ContainerStarted","Data":"8d389cba927a6126f01d9714753bdce5a57cf90ebbee722bf6c95a6200a3302e"}
Jul 19 08:26:16.137023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:16.136926    2559 scope.go:117] "RemoveContainer" containerID="d94f116521bc13ad32ae17bea75a794ca91c0df832760f6f54a3a79f7d63c549"
Jul 19 08:26:17.144968 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:17.144934    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerID="855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363" exitCode=6
Jul 19 08:26:17.145438 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:17.145023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" event={"ID":"cbb66123-374b-4eb0-b5b9-a5abf7ae8796","Type":"ContainerDied","Data":"855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363"}
Jul 19 08:26:17.145438 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:17.145106    2559 scope.go:117] "RemoveContainer" containerID="d94f116521bc13ad32ae17bea75a794ca91c0df832760f6f54a3a79f7d63c549"
Jul 19 08:26:17.145438 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:17.145312    2559 scope.go:117] "RemoveContainer" containerID="855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363"
Jul 19 08:26:17.145594 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:17.145559    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-87h77_odh-ai-gateway-infra(cbb66123-374b-4eb0-b5b9-a5abf7ae8796)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796"
Jul 19 08:26:18.151946 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:18.151922    2559 scope.go:117] "RemoveContainer" containerID="855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363"
Jul 19 08:26:18.152405 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:18.152173    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-87h77_odh-ai-gateway-infra(cbb66123-374b-4eb0-b5b9-a5abf7ae8796)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796"
Jul 19 08:26:29.454618 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:29.454587    2559 scope.go:117] "RemoveContainer" containerID="855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363"
Jul 19 08:26:30.217233 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:30.217202    2559 generic.go:356] "Generic (PLEG): container finished" podID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerID="956c8ce9b2806d08f83e39b7c3a91d933a3b9c332ae21e2ef769e6b8d3abd03c" exitCode=6
Jul 19 08:26:30.217399 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:30.217274    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" event={"ID":"cbb66123-374b-4eb0-b5b9-a5abf7ae8796","Type":"ContainerDied","Data":"956c8ce9b2806d08f83e39b7c3a91d933a3b9c332ae21e2ef769e6b8d3abd03c"}
Jul 19 08:26:30.217399 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:30.217365    2559 scope.go:117] "RemoveContainer" containerID="855a699fba15af01b8d935ab63fd7f4a091e79da5fc348daf07437734ce23363"
Jul 19 08:26:30.217654 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:30.217630    2559 scope.go:117] "RemoveContainer" containerID="956c8ce9b2806d08f83e39b7c3a91d933a3b9c332ae21e2ef769e6b8d3abd03c"
Jul 19 08:26:30.217888 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:30.217868    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-87h77_odh-ai-gateway-infra(cbb66123-374b-4eb0-b5b9-a5abf7ae8796)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796"
Jul 19 08:26:31.241326 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:31.241292    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:31.380283 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:31.380250    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:31.487094 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:31.487065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca\") pod \"cbb66123-374b-4eb0-b5b9-a5abf7ae8796\" (UID: \"cbb66123-374b-4eb0-b5b9-a5abf7ae8796\") "
Jul 19 08:26:31.487446 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:31.487421    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cbb66123-374b-4eb0-b5b9-a5abf7ae8796" (UID: "cbb66123-374b-4eb0-b5b9-a5abf7ae8796"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:26:31.588624 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:31.588592    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb66123-374b-4eb0-b5b9-a5abf7ae8796-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:26:32.229914 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.229884    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"
Jul 19 08:26:32.230091 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.229918    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77" event={"ID":"cbb66123-374b-4eb0-b5b9-a5abf7ae8796","Type":"ContainerDied","Data":"8d389cba927a6126f01d9714753bdce5a57cf90ebbee722bf6c95a6200a3302e"}
Jul 19 08:26:32.230091 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.229977    2559 scope.go:117] "RemoveContainer" containerID="956c8ce9b2806d08f83e39b7c3a91d933a3b9c332ae21e2ef769e6b8d3abd03c"
Jul 19 08:26:32.254557 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.254531    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:32.259688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.259666    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-87h77"]
Jul 19 08:26:32.461292 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:32.461257    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" path="/var/lib/kubelet/pods/cbb66123-374b-4eb0-b5b9-a5abf7ae8796/volumes"
Jul 19 08:26:50.090772 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.090741    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091098    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091109    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091124    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091139    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091145    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091157    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091163    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:50.091215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091220    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091229    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.091537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091234    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8b4b868-50ce-4419-9303-d5c319551229" containerName="revoke-keys"
Jul 19 08:26:50.091537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.091241    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb66123-374b-4eb0-b5b9-a5abf7ae8796" containerName="revoke-keys"
Jul 19 08:26:50.095973 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.095953    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:26:50.098752 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.098729    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:26:50.098872 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.098738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:26:50.101294 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.101267    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:26:50.251909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.251876    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5\" (UID: \"9ebd8f04-a5f7-435d-9724-6313d0d8bb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:26:50.352954 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.352863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5\" (UID: \"9ebd8f04-a5f7-435d-9724-6313d0d8bb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:26:50.353561 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.353541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5\" (UID: \"9ebd8f04-a5f7-435d-9724-6313d0d8bb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:26:50.408911 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.408876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:26:50.539388 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:50.539359    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:26:50.541281 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:26:50.541256    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebd8f04_a5f7_435d_9724_6313d0d8bb53.slice/crio-c8005bd5c7a1e0457130bead75e79cc5496ac92a76cf5d9152e67624dc0eca12 WatchSource:0}: Error finding container c8005bd5c7a1e0457130bead75e79cc5496ac92a76cf5d9152e67624dc0eca12: Status 404 returned error can't find the container with id c8005bd5c7a1e0457130bead75e79cc5496ac92a76cf5d9152e67624dc0eca12
Jul 19 08:26:51.331226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:51.331183    2559 generic.go:356] "Generic (PLEG): container finished" podID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerID="7945db499edc67c515fde2cc27b317063f1b2a26ea3479ad32ad12687979736a" exitCode=6
Jul 19 08:26:51.331226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:51.331223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" event={"ID":"9ebd8f04-a5f7-435d-9724-6313d0d8bb53","Type":"ContainerDied","Data":"7945db499edc67c515fde2cc27b317063f1b2a26ea3479ad32ad12687979736a"}
Jul 19 08:26:51.331662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:51.331267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" event={"ID":"9ebd8f04-a5f7-435d-9724-6313d0d8bb53","Type":"ContainerStarted","Data":"c8005bd5c7a1e0457130bead75e79cc5496ac92a76cf5d9152e67624dc0eca12"}
Jul 19 08:26:51.331662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:51.331441    2559 scope.go:117] "RemoveContainer" containerID="7945db499edc67c515fde2cc27b317063f1b2a26ea3479ad32ad12687979736a"
Jul 19 08:26:52.340413 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:52.340382    2559 generic.go:356] "Generic (PLEG): container finished" podID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerID="95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a" exitCode=6
Jul 19 08:26:52.340928 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:52.340447    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" event={"ID":"9ebd8f04-a5f7-435d-9724-6313d0d8bb53","Type":"ContainerDied","Data":"95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a"}
Jul 19 08:26:52.340928 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:52.340501    2559 scope.go:117] "RemoveContainer" containerID="7945db499edc67c515fde2cc27b317063f1b2a26ea3479ad32ad12687979736a"
Jul 19 08:26:52.340928 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:52.340691    2559 scope.go:117] "RemoveContainer" containerID="95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a"
Jul 19 08:26:52.341106 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:52.340981    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5_odh-ai-gateway-infra(9ebd8f04-a5f7-435d-9724-6313d0d8bb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53"
Jul 19 08:26:53.347029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:26:53.347001    2559 scope.go:117] "RemoveContainer" containerID="95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a"
Jul 19 08:26:53.347416 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:26:53.347198    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5_odh-ai-gateway-infra(9ebd8f04-a5f7-435d-9724-6313d0d8bb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53"
Jul 19 08:27:06.454718 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:06.454674    2559 scope.go:117] "RemoveContainer" containerID="95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a"
Jul 19 08:27:07.421601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:07.421516    2559 generic.go:356] "Generic (PLEG): container finished" podID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerID="ec4ec9dd6337dfd89aafc44e208e56a38cdc882785e1b31d7160e63c35df6c9a" exitCode=6
Jul 19 08:27:07.421601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:07.421585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" event={"ID":"9ebd8f04-a5f7-435d-9724-6313d0d8bb53","Type":"ContainerDied","Data":"ec4ec9dd6337dfd89aafc44e208e56a38cdc882785e1b31d7160e63c35df6c9a"}
Jul 19 08:27:07.421817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:07.421670    2559 scope.go:117] "RemoveContainer" containerID="95ed448b37fc61d5fa501fd5fa1e0d5685f1b5c68b6651bc1e1849539ad8c12a"
Jul 19 08:27:07.422001 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:07.421981    2559 scope.go:117] "RemoveContainer" containerID="ec4ec9dd6337dfd89aafc44e208e56a38cdc882785e1b31d7160e63c35df6c9a"
Jul 19 08:27:07.422246 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:27:07.422224    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5_odh-ai-gateway-infra(9ebd8f04-a5f7-435d-9724-6313d0d8bb53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53"
Jul 19 08:27:08.449145 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:08.449109    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:27:08.592151 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:08.592117    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:27:08.718212 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:08.718126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca\") pod \"9ebd8f04-a5f7-435d-9724-6313d0d8bb53\" (UID: \"9ebd8f04-a5f7-435d-9724-6313d0d8bb53\") "
Jul 19 08:27:08.718489 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:08.718467    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ebd8f04-a5f7-435d-9724-6313d0d8bb53" (UID: "9ebd8f04-a5f7-435d-9724-6313d0d8bb53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:27:08.819779 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:08.819744    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebd8f04-a5f7-435d-9724-6313d0d8bb53-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:27:09.434812 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:09.434779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5" event={"ID":"9ebd8f04-a5f7-435d-9724-6313d0d8bb53","Type":"ContainerDied","Data":"c8005bd5c7a1e0457130bead75e79cc5496ac92a76cf5d9152e67624dc0eca12"}
Jul 19 08:27:09.434812 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:09.434799    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"
Jul 19 08:27:09.435076 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:09.434836    2559 scope.go:117] "RemoveContainer" containerID="ec4ec9dd6337dfd89aafc44e208e56a38cdc882785e1b31d7160e63c35df6c9a"
Jul 19 08:27:09.455197 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:09.455171    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:27:09.458386 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:09.458362    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7f6h5"]
Jul 19 08:27:10.460765 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:10.460729    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" path="/var/lib/kubelet/pods/9ebd8f04-a5f7-435d-9724-6313d0d8bb53/volumes"
Jul 19 08:27:40.130941 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.130910    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131287    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131298    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131316    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131323    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131330    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131335    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131400    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131408    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.131441 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.131415    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebd8f04-a5f7-435d-9724-6313d0d8bb53" containerName="revoke-keys"
Jul 19 08:27:40.134442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.134426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:40.136902 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.136881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:27:40.137025 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.136913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:27:40.144312 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.144281    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:27:40.206177 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.206146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp\" (UID: \"d7690c7a-e7ae-49ce-9e4e-4d373e769578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:40.307663 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.307627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp\" (UID: \"d7690c7a-e7ae-49ce-9e4e-4d373e769578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:40.308228 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.308209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp\" (UID: \"d7690c7a-e7ae-49ce-9e4e-4d373e769578\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:40.450432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.450348    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:40.581117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.581092    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:27:40.583098 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:27:40.583072    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7690c7a_e7ae_49ce_9e4e_4d373e769578.slice/crio-5502d76b2a8d0c322d738bd796da723dabe34c90d9e3960b86caa66a56ff0ad1 WatchSource:0}: Error finding container 5502d76b2a8d0c322d738bd796da723dabe34c90d9e3960b86caa66a56ff0ad1: Status 404 returned error can't find the container with id 5502d76b2a8d0c322d738bd796da723dabe34c90d9e3960b86caa66a56ff0ad1
Jul 19 08:27:40.602644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:40.602621    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" event={"ID":"d7690c7a-e7ae-49ce-9e4e-4d373e769578","Type":"ContainerStarted","Data":"5502d76b2a8d0c322d738bd796da723dabe34c90d9e3960b86caa66a56ff0ad1"}
Jul 19 08:27:41.609488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:41.609455    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerID="3ba5f8952849d0c982e422b7150d743d1cc197df02f2748643381ef9cbe4f55f" exitCode=6
Jul 19 08:27:41.609911 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:41.609508    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" event={"ID":"d7690c7a-e7ae-49ce-9e4e-4d373e769578","Type":"ContainerDied","Data":"3ba5f8952849d0c982e422b7150d743d1cc197df02f2748643381ef9cbe4f55f"}
Jul 19 08:27:41.609911 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:41.609778    2559 scope.go:117] "RemoveContainer" containerID="3ba5f8952849d0c982e422b7150d743d1cc197df02f2748643381ef9cbe4f55f"
Jul 19 08:27:42.616454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:42.616421    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerID="83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8" exitCode=6
Jul 19 08:27:42.616885 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:42.616509    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" event={"ID":"d7690c7a-e7ae-49ce-9e4e-4d373e769578","Type":"ContainerDied","Data":"83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8"}
Jul 19 08:27:42.616885 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:42.616566    2559 scope.go:117] "RemoveContainer" containerID="3ba5f8952849d0c982e422b7150d743d1cc197df02f2748643381ef9cbe4f55f"
Jul 19 08:27:42.616885 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:42.616696    2559 scope.go:117] "RemoveContainer" containerID="83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8"
Jul 19 08:27:42.617013 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:27:42.616969    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp_odh-ai-gateway-infra(d7690c7a-e7ae-49ce-9e4e-4d373e769578)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578"
Jul 19 08:27:43.623830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:43.623802    2559 scope.go:117] "RemoveContainer" containerID="83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8"
Jul 19 08:27:43.624208 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:27:43.623985    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp_odh-ai-gateway-infra(d7690c7a-e7ae-49ce-9e4e-4d373e769578)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578"
Jul 19 08:27:55.455674 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:55.455647    2559 scope.go:117] "RemoveContainer" containerID="83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8"
Jul 19 08:27:56.700159 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:56.700127    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerID="6650c619fdea7c9124c629e98d293076987a500653b06a7da2522597aeaabb3c" exitCode=6
Jul 19 08:27:56.700642 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:56.700192    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" event={"ID":"d7690c7a-e7ae-49ce-9e4e-4d373e769578","Type":"ContainerDied","Data":"6650c619fdea7c9124c629e98d293076987a500653b06a7da2522597aeaabb3c"}
Jul 19 08:27:56.700642 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:56.700249    2559 scope.go:117] "RemoveContainer" containerID="83e1be5e9ae088d5a053a7f96b55f66562e10d2574982d1ccfed57f5fbc0f3e8"
Jul 19 08:27:56.700642 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:56.700570    2559 scope.go:117] "RemoveContainer" containerID="6650c619fdea7c9124c629e98d293076987a500653b06a7da2522597aeaabb3c"
Jul 19 08:27:56.700878 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:27:56.700856    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp_odh-ai-gateway-infra(d7690c7a-e7ae-49ce-9e4e-4d373e769578)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578"
Jul 19 08:27:57.724425 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:57.724391    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:27:57.860017 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:57.859994    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:57.961015 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:57.960983    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca\") pod \"d7690c7a-e7ae-49ce-9e4e-4d373e769578\" (UID: \"d7690c7a-e7ae-49ce-9e4e-4d373e769578\") "
Jul 19 08:27:57.961358 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:57.961339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7690c7a-e7ae-49ce-9e4e-4d373e769578" (UID: "d7690c7a-e7ae-49ce-9e4e-4d373e769578"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:27:58.062411 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.062379    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7690c7a-e7ae-49ce-9e4e-4d373e769578-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:27:58.713447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.713423    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"
Jul 19 08:27:58.713617 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.713464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp" event={"ID":"d7690c7a-e7ae-49ce-9e4e-4d373e769578","Type":"ContainerDied","Data":"5502d76b2a8d0c322d738bd796da723dabe34c90d9e3960b86caa66a56ff0ad1"}
Jul 19 08:27:58.713617 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.713529    2559 scope.go:117] "RemoveContainer" containerID="6650c619fdea7c9124c629e98d293076987a500653b06a7da2522597aeaabb3c"
Jul 19 08:27:58.734420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.734394    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:27:58.736547 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:27:58.736528    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-28fvp"]
Jul 19 08:28:00.460765 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:00.460731    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" path="/var/lib/kubelet/pods/d7690c7a-e7ae-49ce-9e4e-4d373e769578/volumes"
Jul 19 08:28:05.094225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094190    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094530    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094540    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094545    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094558    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094564    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094818 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094624    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094818 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094632    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.094818 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.094639    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7690c7a-e7ae-49ce-9e4e-4d373e769578" containerName="revoke-keys"
Jul 19 08:28:05.098992 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.098975    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:05.101849 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.101825    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:28:05.102024 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.101825    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:28:05.105985 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.105962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:05.227878 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.227842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r\" (UID: \"54e8f074-e480-41dd-8fb0-f6cb2198ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:05.328359 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.328320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r\" (UID: \"54e8f074-e480-41dd-8fb0-f6cb2198ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:05.329024 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.329006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r\" (UID: \"54e8f074-e480-41dd-8fb0-f6cb2198ffa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:05.411660 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.411577    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:05.544075 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.541091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:05.547060 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:28:05.547028    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e8f074_e480_41dd_8fb0_f6cb2198ffa3.slice/crio-62d0984d08bcc6937a91975d31c790aa8ea37ba0eb8ebab7e8c2d1993844ebe3 WatchSource:0}: Error finding container 62d0984d08bcc6937a91975d31c790aa8ea37ba0eb8ebab7e8c2d1993844ebe3: Status 404 returned error can't find the container with id 62d0984d08bcc6937a91975d31c790aa8ea37ba0eb8ebab7e8c2d1993844ebe3
Jul 19 08:28:05.751454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:05.751408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerStarted","Data":"62d0984d08bcc6937a91975d31c790aa8ea37ba0eb8ebab7e8c2d1993844ebe3"}
Jul 19 08:28:06.269848 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.269813    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"]
Jul 19 08:28:06.273200 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.273185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:06.279901 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.279878    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"]
Jul 19 08:28:06.338160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.338128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw\" (UID: \"48036c07-85c4-4084-bed3-dacec8924361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:06.438759 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.438666    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw\" (UID: \"48036c07-85c4-4084-bed3-dacec8924361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:06.439226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.439208    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw\" (UID: \"48036c07-85c4-4084-bed3-dacec8924361\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:06.585170 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.585133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:06.714318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.714288    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"]
Jul 19 08:28:06.715518 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:28:06.715496    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48036c07_85c4_4084_bed3_dacec8924361.slice/crio-7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735 WatchSource:0}: Error finding container 7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735: Status 404 returned error can't find the container with id 7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735
Jul 19 08:28:06.759462 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.759435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw" event={"ID":"48036c07-85c4-4084-bed3-dacec8924361","Type":"ContainerStarted","Data":"7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735"}
Jul 19 08:28:06.760868 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.760845    2559 generic.go:356] "Generic (PLEG): container finished" podID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerID="617828d477c8f9f9bf46c3ed7938021d966f38c5a370a12d931751895ade733a" exitCode=6
Jul 19 08:28:06.760982 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.760900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerDied","Data":"617828d477c8f9f9bf46c3ed7938021d966f38c5a370a12d931751895ade733a"}
Jul 19 08:28:06.761132 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:06.761115    2559 scope.go:117] "RemoveContainer" containerID="617828d477c8f9f9bf46c3ed7938021d966f38c5a370a12d931751895ade733a"
Jul 19 08:28:07.769613 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.769523    2559 generic.go:356] "Generic (PLEG): container finished" podID="48036c07-85c4-4084-bed3-dacec8924361" containerID="fedbff6c8bdab6593770f13e22986059759a91517f6437964f74672e6a889ab0" exitCode=0
Jul 19 08:28:07.770053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.769611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw" event={"ID":"48036c07-85c4-4084-bed3-dacec8924361","Type":"ContainerDied","Data":"fedbff6c8bdab6593770f13e22986059759a91517f6437964f74672e6a889ab0"}
Jul 19 08:28:07.771355 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.771334    2559 generic.go:356] "Generic (PLEG): container finished" podID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerID="11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868" exitCode=6
Jul 19 08:28:07.771481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.771418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerDied","Data":"11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868"}
Jul 19 08:28:07.771481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.771467    2559 scope.go:117] "RemoveContainer" containerID="617828d477c8f9f9bf46c3ed7938021d966f38c5a370a12d931751895ade733a"
Jul 19 08:28:07.771655 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:07.771636    2559 scope.go:117] "RemoveContainer" containerID="11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868"
Jul 19 08:28:07.771881 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:07.771863    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r_odh-ai-gateway-infra(54e8f074-e480-41dd-8fb0-f6cb2198ffa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3"
Jul 19 08:28:08.778481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:08.778452    2559 scope.go:117] "RemoveContainer" containerID="11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868"
Jul 19 08:28:08.778900 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:08.778688    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r_odh-ai-gateway-infra(54e8f074-e480-41dd-8fb0-f6cb2198ffa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3"
Jul 19 08:28:08.910314 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:08.910290    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:09.066964 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.066922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca\") pod \"48036c07-85c4-4084-bed3-dacec8924361\" (UID: \"48036c07-85c4-4084-bed3-dacec8924361\") "
Jul 19 08:28:09.067279 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.067257    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48036c07-85c4-4084-bed3-dacec8924361" (UID: "48036c07-85c4-4084-bed3-dacec8924361"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:28:09.168124 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.168095    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48036c07-85c4-4084-bed3-dacec8924361-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:28:09.784010 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.783982    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw"
Jul 19 08:28:09.784418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.783982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw" event={"ID":"48036c07-85c4-4084-bed3-dacec8924361","Type":"ContainerDied","Data":"7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735"}
Jul 19 08:28:09.784418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:09.784127    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7348d2bd02e04cab6ae251efa5edfb55e54de3d1b50cc77d8fd5410004ddb735"
Jul 19 08:28:24.457494 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:24.457463    2559 scope.go:117] "RemoveContainer" containerID="11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868"
Jul 19 08:28:24.865112 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:24.865080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerStarted","Data":"85567ec8063bc7e91c27a25af6bc189e7e3d2b9df69c02578fc94522fc1d7e19"}
Jul 19 08:28:24.880315 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:24.880263    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" podStartSLOduration=19.880248214 podStartE2EDuration="19.880248214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:28:24.879196048 +0000 UTC m=+2741.388793947" watchObservedRunningTime="2026-07-19 08:28:24.880248214 +0000 UTC m=+2741.389846113"
Jul 19 08:28:25.872142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:25.872106    2559 generic.go:356] "Generic (PLEG): container finished" podID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerID="85567ec8063bc7e91c27a25af6bc189e7e3d2b9df69c02578fc94522fc1d7e19" exitCode=6
Jul 19 08:28:25.872142 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:25.872145    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerDied","Data":"85567ec8063bc7e91c27a25af6bc189e7e3d2b9df69c02578fc94522fc1d7e19"}
Jul 19 08:28:25.872673 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:25.872194    2559 scope.go:117] "RemoveContainer" containerID="11c3b286aa62725f3779ba30b28359243b206b151c55a134a09c61549987c868"
Jul 19 08:28:25.872673 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:25.872502    2559 scope.go:117] "RemoveContainer" containerID="85567ec8063bc7e91c27a25af6bc189e7e3d2b9df69c02578fc94522fc1d7e19"
Jul 19 08:28:25.872815 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:25.872759    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r_odh-ai-gateway-infra(54e8f074-e480-41dd-8fb0-f6cb2198ffa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3"
Jul 19 08:28:25.895295 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:25.895267    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:27.011496 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.011475    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:27.043161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.043129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca\") pod \"54e8f074-e480-41dd-8fb0-f6cb2198ffa3\" (UID: \"54e8f074-e480-41dd-8fb0-f6cb2198ffa3\") "
Jul 19 08:28:27.043565 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.043539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "54e8f074-e480-41dd-8fb0-f6cb2198ffa3" (UID: "54e8f074-e480-41dd-8fb0-f6cb2198ffa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:28:27.144233 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.144156    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e8f074-e480-41dd-8fb0-f6cb2198ffa3-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:28:27.884780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.884750    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"
Jul 19 08:28:27.884958 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.884749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r" event={"ID":"54e8f074-e480-41dd-8fb0-f6cb2198ffa3","Type":"ContainerDied","Data":"62d0984d08bcc6937a91975d31c790aa8ea37ba0eb8ebab7e8c2d1993844ebe3"}
Jul 19 08:28:27.884958 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.884883    2559 scope.go:117] "RemoveContainer" containerID="85567ec8063bc7e91c27a25af6bc189e7e3d2b9df69c02578fc94522fc1d7e19"
Jul 19 08:28:27.915225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.915194    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:27.918794 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:27.918771    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-24r4r"]
Jul 19 08:28:28.460224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:28.460195    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" path="/var/lib/kubelet/pods/54e8f074-e480-41dd-8fb0-f6cb2198ffa3/volumes"
Jul 19 08:28:30.172817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.172780    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48036c07-85c4-4084-bed3-dacec8924361" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173158    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48036c07-85c4-4084-bed3-dacec8924361" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173171    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173176    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173187    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173198    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173511 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173262    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.173511 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173269    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48036c07-85c4-4084-bed3-dacec8924361" containerName="revoke-keys"
Jul 19 08:28:30.173511 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.173278    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:28:30.177751 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.177723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:30.180843 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.180818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:28:30.180980 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.180823    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:28:30.183166 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.183139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:30.270519 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.270482    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8\" (UID: \"8a332275-46d2-4e89-9edf-421e76141e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:30.371423 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.371393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8\" (UID: \"8a332275-46d2-4e89-9edf-421e76141e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:30.372020 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.372000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8\" (UID: \"8a332275-46d2-4e89-9edf-421e76141e25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:30.491447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.491354    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:30.829592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.829523    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:30.831833 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:28:30.831807    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a332275_46d2_4e89_9edf_421e76141e25.slice/crio-7d097652d3d66a4cfd98900d2825ff555f07648c8686080ab4838a74989a5cc3 WatchSource:0}: Error finding container 7d097652d3d66a4cfd98900d2825ff555f07648c8686080ab4838a74989a5cc3: Status 404 returned error can't find the container with id 7d097652d3d66a4cfd98900d2825ff555f07648c8686080ab4838a74989a5cc3
Jul 19 08:28:30.903222 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:30.903198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerStarted","Data":"7d097652d3d66a4cfd98900d2825ff555f07648c8686080ab4838a74989a5cc3"}
Jul 19 08:28:31.909860 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:31.909827    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a332275-46d2-4e89-9edf-421e76141e25" containerID="f61e71b97b7b3864595a830fee813b3b6da37d566a8eb733ffa30521eae0f408" exitCode=6
Jul 19 08:28:31.910246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:31.909891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerDied","Data":"f61e71b97b7b3864595a830fee813b3b6da37d566a8eb733ffa30521eae0f408"}
Jul 19 08:28:31.910246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:31.910097    2559 scope.go:117] "RemoveContainer" containerID="f61e71b97b7b3864595a830fee813b3b6da37d566a8eb733ffa30521eae0f408"
Jul 19 08:28:32.917419 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:32.917388    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a332275-46d2-4e89-9edf-421e76141e25" containerID="f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1" exitCode=6
Jul 19 08:28:32.917942 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:32.917482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerDied","Data":"f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1"}
Jul 19 08:28:32.917942 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:32.917552    2559 scope.go:117] "RemoveContainer" containerID="f61e71b97b7b3864595a830fee813b3b6da37d566a8eb733ffa30521eae0f408"
Jul 19 08:28:32.917942 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:32.917769    2559 scope.go:117] "RemoveContainer" containerID="f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1"
Jul 19 08:28:32.918076 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:32.918005    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8_odh-ai-gateway-infra(8a332275-46d2-4e89-9edf-421e76141e25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" podUID="8a332275-46d2-4e89-9edf-421e76141e25"
Jul 19 08:28:33.924683 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:33.924655    2559 scope.go:117] "RemoveContainer" containerID="f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1"
Jul 19 08:28:33.925085 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:33.924881    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8_odh-ai-gateway-infra(8a332275-46d2-4e89-9edf-421e76141e25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" podUID="8a332275-46d2-4e89-9edf-421e76141e25"
Jul 19 08:28:46.455322 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:46.455285    2559 scope.go:117] "RemoveContainer" containerID="f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1"
Jul 19 08:28:46.995923 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:46.995888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerStarted","Data":"40b2e7c49b3a5cb2847eb0d05957304bcbcbf16c2adc4199197d032c34ad3b23"}
Jul 19 08:28:47.012060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:47.012013    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" podStartSLOduration=17.011996733 podStartE2EDuration="17.011996733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:28:47.008928764 +0000 UTC m=+2763.518526663" watchObservedRunningTime="2026-07-19 08:28:47.011996733 +0000 UTC m=+2763.521594632"
Jul 19 08:28:48.002726 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:48.002672    2559 generic.go:356] "Generic (PLEG): container finished" podID="8a332275-46d2-4e89-9edf-421e76141e25" containerID="40b2e7c49b3a5cb2847eb0d05957304bcbcbf16c2adc4199197d032c34ad3b23" exitCode=6
Jul 19 08:28:48.003216 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:48.002746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerDied","Data":"40b2e7c49b3a5cb2847eb0d05957304bcbcbf16c2adc4199197d032c34ad3b23"}
Jul 19 08:28:48.003216 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:48.002808    2559 scope.go:117] "RemoveContainer" containerID="f14fbaad9933f745b6dffd93bff0724a8518afa38864d0a864cc27f9966e6fc1"
Jul 19 08:28:48.003216 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:48.003136    2559 scope.go:117] "RemoveContainer" containerID="40b2e7c49b3a5cb2847eb0d05957304bcbcbf16c2adc4199197d032c34ad3b23"
Jul 19 08:28:48.003428 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:28:48.003406    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8_odh-ai-gateway-infra(8a332275-46d2-4e89-9edf-421e76141e25)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" podUID="8a332275-46d2-4e89-9edf-421e76141e25"
Jul 19 08:28:48.026352 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:48.026327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:49.147260 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:49.147233    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:49.243537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:49.243500    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca\") pod \"8a332275-46d2-4e89-9edf-421e76141e25\" (UID: \"8a332275-46d2-4e89-9edf-421e76141e25\") "
Jul 19 08:28:49.243901 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:49.243876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a332275-46d2-4e89-9edf-421e76141e25" (UID: "8a332275-46d2-4e89-9edf-421e76141e25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:28:49.344679 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:49.344646    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a332275-46d2-4e89-9edf-421e76141e25-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:28:50.015325 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.015292    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"
Jul 19 08:28:50.015325 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.015315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8" event={"ID":"8a332275-46d2-4e89-9edf-421e76141e25","Type":"ContainerDied","Data":"7d097652d3d66a4cfd98900d2825ff555f07648c8686080ab4838a74989a5cc3"}
Jul 19 08:28:50.015544 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.015377    2559 scope.go:117] "RemoveContainer" containerID="40b2e7c49b3a5cb2847eb0d05957304bcbcbf16c2adc4199197d032c34ad3b23"
Jul 19 08:28:50.037104 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.037079    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:50.042492 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.042458    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-94zp8"]
Jul 19 08:28:50.461508 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:28:50.461474    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a332275-46d2-4e89-9edf-421e76141e25" path="/var/lib/kubelet/pods/8a332275-46d2-4e89-9edf-421e76141e25/volumes"
Jul 19 08:29:37.463248 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463215    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463578    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463590    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463615    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463622    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463679    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e8f074-e480-41dd-8fb0-f6cb2198ffa3" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463686    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.463737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.463693    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:29:37.466660 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.466644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:37.469029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.469003    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:29:37.469029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.469014    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:29:37.474024 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.473995    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:37.489106 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.489078    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2\" (UID: \"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:37.590381 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.590345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2\" (UID: \"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:37.591039 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.591011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2\" (UID: \"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:37.780215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.780110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:37.912184 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:37.912158    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:37.913832 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:29:37.913801    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe4c5dc_30ee_4c4c_aeda_88c763f0c75a.slice/crio-3a128199affd5efb329bd701d75c82514b91103dd71b44707a8b1c95451652ff WatchSource:0}: Error finding container 3a128199affd5efb329bd701d75c82514b91103dd71b44707a8b1c95451652ff: Status 404 returned error can't find the container with id 3a128199affd5efb329bd701d75c82514b91103dd71b44707a8b1c95451652ff
Jul 19 08:29:38.269847 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:38.269814    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerStarted","Data":"9aadb8126817c5363032f84675ec03b88cda54b0c703a00960bc2628d5473eeb"}
Jul 19 08:29:38.270019 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:38.269863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerStarted","Data":"3a128199affd5efb329bd701d75c82514b91103dd71b44707a8b1c95451652ff"}
Jul 19 08:29:38.283963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:38.283913    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" podStartSLOduration=1.283893569 podStartE2EDuration="1.283893569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:29:38.283263262 +0000 UTC m=+2814.792861162" watchObservedRunningTime="2026-07-19 08:29:38.283893569 +0000 UTC m=+2814.793491470"
Jul 19 08:29:39.276330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:39.276294    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerID="9aadb8126817c5363032f84675ec03b88cda54b0c703a00960bc2628d5473eeb" exitCode=6
Jul 19 08:29:39.276843 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:39.276385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerDied","Data":"9aadb8126817c5363032f84675ec03b88cda54b0c703a00960bc2628d5473eeb"}
Jul 19 08:29:39.276843 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:39.276583    2559 scope.go:117] "RemoveContainer" containerID="9aadb8126817c5363032f84675ec03b88cda54b0c703a00960bc2628d5473eeb"
Jul 19 08:29:40.283722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:40.283615    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerID="dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec" exitCode=6
Jul 19 08:29:40.283722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:40.283710    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerDied","Data":"dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec"}
Jul 19 08:29:40.284229 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:40.283766    2559 scope.go:117] "RemoveContainer" containerID="9aadb8126817c5363032f84675ec03b88cda54b0c703a00960bc2628d5473eeb"
Jul 19 08:29:40.284229 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:40.283983    2559 scope.go:117] "RemoveContainer" containerID="dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec"
Jul 19 08:29:40.284229 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:29:40.284216    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2_odh-ai-gateway-infra(1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a"
Jul 19 08:29:41.290892 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:41.290864    2559 scope.go:117] "RemoveContainer" containerID="dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec"
Jul 19 08:29:41.291269 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:29:41.291079    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2_odh-ai-gateway-infra(1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a"
Jul 19 08:29:54.457660 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:54.457626    2559 scope.go:117] "RemoveContainer" containerID="dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec"
Jul 19 08:29:55.365588 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:55.365508    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerID="a8cdc6b550ac08b819be9bcef9275a89bfd87cf2abe9582a8cd2388422107a89" exitCode=6
Jul 19 08:29:55.365747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:55.365588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerDied","Data":"a8cdc6b550ac08b819be9bcef9275a89bfd87cf2abe9582a8cd2388422107a89"}
Jul 19 08:29:55.365747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:55.365647    2559 scope.go:117] "RemoveContainer" containerID="dcbe7e4437d75f3f5142134334dcf4c7776e4d2519401fcd40573c0dcb520dec"
Jul 19 08:29:55.366022 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:55.366007    2559 scope.go:117] "RemoveContainer" containerID="a8cdc6b550ac08b819be9bcef9275a89bfd87cf2abe9582a8cd2388422107a89"
Jul 19 08:29:55.366238 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:29:55.366211    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2_odh-ai-gateway-infra(1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a"
Jul 19 08:29:56.390236 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:56.390205    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:56.523179 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:56.523158    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:56.659603 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:56.659507    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca\") pod \"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a\" (UID: \"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a\") "
Jul 19 08:29:56.659892 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:56.659866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" (UID: "1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:29:56.760854 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:56.760816    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:29:57.379172 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:57.379142    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"
Jul 19 08:29:57.379372 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:57.379178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2" event={"ID":"1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a","Type":"ContainerDied","Data":"3a128199affd5efb329bd701d75c82514b91103dd71b44707a8b1c95451652ff"}
Jul 19 08:29:57.379372 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:57.379236    2559 scope.go:117] "RemoveContainer" containerID="a8cdc6b550ac08b819be9bcef9275a89bfd87cf2abe9582a8cd2388422107a89"
Jul 19 08:29:57.401649 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:57.401625    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:57.407082 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:57.407058    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-rcqz2"]
Jul 19 08:29:58.461094 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:29:58.461060    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" path="/var/lib/kubelet/pods/1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a/volumes"
Jul 19 08:30:23.454038 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454001    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454339    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454350    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454380    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454385    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454390    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454399    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454404    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454458    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454465    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a332275-46d2-4e89-9edf-421e76141e25" containerName="revoke-keys"
Jul 19 08:30:23.454552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.454472    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:30:23.457220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.457204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:23.460955 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.460934    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:30:23.460955 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.460948    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:30:23.465961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.465938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:23.501454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.501419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf\" (UID: \"eef236c4-400b-4621-adef-7d0612005b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:23.602037 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.602002    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf\" (UID: \"eef236c4-400b-4621-adef-7d0612005b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:23.602615 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.602596    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf\" (UID: \"eef236c4-400b-4621-adef-7d0612005b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:23.768542 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.768447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:23.896797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:23.896770    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:23.898163 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:30:23.898138    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef236c4_400b_4621_adef_7d0612005b45.slice/crio-c16241f8a1ee3cf45cda91c0406ed75cc12f14a151012c48e8231e20fb85843b WatchSource:0}: Error finding container c16241f8a1ee3cf45cda91c0406ed75cc12f14a151012c48e8231e20fb85843b: Status 404 returned error can't find the container with id c16241f8a1ee3cf45cda91c0406ed75cc12f14a151012c48e8231e20fb85843b
Jul 19 08:30:24.521095 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:24.521067    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef236c4-400b-4621-adef-7d0612005b45" containerID="375bcc310068f798792b7401e72da81c3364766b24420c720cb726fa9bf96953" exitCode=6
Jul 19 08:30:24.521418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:24.521139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" event={"ID":"eef236c4-400b-4621-adef-7d0612005b45","Type":"ContainerDied","Data":"375bcc310068f798792b7401e72da81c3364766b24420c720cb726fa9bf96953"}
Jul 19 08:30:24.521418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:24.521184    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" event={"ID":"eef236c4-400b-4621-adef-7d0612005b45","Type":"ContainerStarted","Data":"c16241f8a1ee3cf45cda91c0406ed75cc12f14a151012c48e8231e20fb85843b"}
Jul 19 08:30:24.521418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:24.521288    2559 scope.go:117] "RemoveContainer" containerID="375bcc310068f798792b7401e72da81c3364766b24420c720cb726fa9bf96953"
Jul 19 08:30:25.528582 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:25.528494    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef236c4-400b-4621-adef-7d0612005b45" containerID="d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6" exitCode=6
Jul 19 08:30:25.528582 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:25.528536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" event={"ID":"eef236c4-400b-4621-adef-7d0612005b45","Type":"ContainerDied","Data":"d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6"}
Jul 19 08:30:25.529086 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:25.528595    2559 scope.go:117] "RemoveContainer" containerID="375bcc310068f798792b7401e72da81c3364766b24420c720cb726fa9bf96953"
Jul 19 08:30:25.529086 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:25.528808    2559 scope.go:117] "RemoveContainer" containerID="d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6"
Jul 19 08:30:25.529086 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:30:25.528995    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf_odh-ai-gateway-infra(eef236c4-400b-4621-adef-7d0612005b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" podUID="eef236c4-400b-4621-adef-7d0612005b45"
Jul 19 08:30:26.535984 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:26.535957    2559 scope.go:117] "RemoveContainer" containerID="d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6"
Jul 19 08:30:26.536470 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:30:26.536186    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf_odh-ai-gateway-infra(eef236c4-400b-4621-adef-7d0612005b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" podUID="eef236c4-400b-4621-adef-7d0612005b45"
Jul 19 08:30:38.454948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:38.454869    2559 scope.go:117] "RemoveContainer" containerID="d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6"
Jul 19 08:30:39.608925 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:39.608893    2559 generic.go:356] "Generic (PLEG): container finished" podID="eef236c4-400b-4621-adef-7d0612005b45" containerID="a9a5452eea2bee91087203ff8f805f91e64173e50624154228d3d143c9cba601" exitCode=6
Jul 19 08:30:39.609330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:39.608960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" event={"ID":"eef236c4-400b-4621-adef-7d0612005b45","Type":"ContainerDied","Data":"a9a5452eea2bee91087203ff8f805f91e64173e50624154228d3d143c9cba601"}
Jul 19 08:30:39.609330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:39.609024    2559 scope.go:117] "RemoveContainer" containerID="d3c54f1406708e6f4becf6ab781c47ae0607a1f70801f8efa948b7f98a7824e6"
Jul 19 08:30:39.609330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:39.609294    2559 scope.go:117] "RemoveContainer" containerID="a9a5452eea2bee91087203ff8f805f91e64173e50624154228d3d143c9cba601"
Jul 19 08:30:39.609541 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:30:39.609522    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf_odh-ai-gateway-infra(eef236c4-400b-4621-adef-7d0612005b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" podUID="eef236c4-400b-4621-adef-7d0612005b45"
Jul 19 08:30:40.633526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:40.633495    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:40.780562 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:40.780538    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:40.853775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:40.853745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca\") pod \"eef236c4-400b-4621-adef-7d0612005b45\" (UID: \"eef236c4-400b-4621-adef-7d0612005b45\") "
Jul 19 08:30:40.854060 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:40.854037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eef236c4-400b-4621-adef-7d0612005b45" (UID: "eef236c4-400b-4621-adef-7d0612005b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:30:40.955255 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:40.955171    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef236c4-400b-4621-adef-7d0612005b45-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:30:41.622767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:41.622738    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"
Jul 19 08:30:41.622767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:41.622740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf" event={"ID":"eef236c4-400b-4621-adef-7d0612005b45","Type":"ContainerDied","Data":"c16241f8a1ee3cf45cda91c0406ed75cc12f14a151012c48e8231e20fb85843b"}
Jul 19 08:30:41.623029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:41.622805    2559 scope.go:117] "RemoveContainer" containerID="a9a5452eea2bee91087203ff8f805f91e64173e50624154228d3d143c9cba601"
Jul 19 08:30:41.645577 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:41.645548    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:41.652733 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:41.652710    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-g65bf"]
Jul 19 08:30:42.461543 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:30:42.461509    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef236c4-400b-4621-adef-7d0612005b45" path="/var/lib/kubelet/pods/eef236c4-400b-4621-adef-7d0612005b45/volumes"
Jul 19 08:31:05.237365 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.237326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:05.237978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.237898    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.237978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.237919    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.237978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.237934    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.237978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.237943    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.238202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.238051    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.238202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.238062    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe4c5dc-30ee-4c4c-aeda-88c763f0c75a" containerName="revoke-keys"
Jul 19 08:31:05.238202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.238074    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.238202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.238085    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:05.241190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.241170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:05.243647 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.243627    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:31:05.243819 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.243799    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:31:05.247903 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.247882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:05.382513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.382475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5\" (UID: \"c3edd927-4b9b-4956-945c-057a96ba0229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:05.483926 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.483885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5\" (UID: \"c3edd927-4b9b-4956-945c-057a96ba0229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:05.484539 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.484517    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5\" (UID: \"c3edd927-4b9b-4956-945c-057a96ba0229\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:05.554806 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.554694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:05.891809 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:05.891778    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:05.893891 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:05.893859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3edd927_4b9b_4956_945c_057a96ba0229.slice/crio-3cfa661ee092c63486c90c8aa025b33d1af539bbeec28348199ac59f013c536d WatchSource:0}: Error finding container 3cfa661ee092c63486c90c8aa025b33d1af539bbeec28348199ac59f013c536d: Status 404 returned error can't find the container with id 3cfa661ee092c63486c90c8aa025b33d1af539bbeec28348199ac59f013c536d
Jul 19 08:31:06.764328 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:06.764232    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3edd927-4b9b-4956-945c-057a96ba0229" containerID="7efcb6a7ef49e3ac70d989d926a1d3785d3570741da114d5f5cec584a1a0036e" exitCode=6
Jul 19 08:31:06.764328 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:06.764267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerDied","Data":"7efcb6a7ef49e3ac70d989d926a1d3785d3570741da114d5f5cec584a1a0036e"}
Jul 19 08:31:06.764328 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:06.764316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerStarted","Data":"3cfa661ee092c63486c90c8aa025b33d1af539bbeec28348199ac59f013c536d"}
Jul 19 08:31:06.764846 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:06.764500    2559 scope.go:117] "RemoveContainer" containerID="7efcb6a7ef49e3ac70d989d926a1d3785d3570741da114d5f5cec584a1a0036e"
Jul 19 08:31:07.771467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:07.771382    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3edd927-4b9b-4956-945c-057a96ba0229" containerID="9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca" exitCode=6
Jul 19 08:31:07.771898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:07.771472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerDied","Data":"9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca"}
Jul 19 08:31:07.771898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:07.771531    2559 scope.go:117] "RemoveContainer" containerID="7efcb6a7ef49e3ac70d989d926a1d3785d3570741da114d5f5cec584a1a0036e"
Jul 19 08:31:07.771898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:07.771647    2559 scope.go:117] "RemoveContainer" containerID="9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca"
Jul 19 08:31:07.771898 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:07.771874    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5_odh-ai-gateway-infra(c3edd927-4b9b-4956-945c-057a96ba0229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" podUID="c3edd927-4b9b-4956-945c-057a96ba0229"
Jul 19 08:31:08.778454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:08.778429    2559 scope.go:117] "RemoveContainer" containerID="9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca"
Jul 19 08:31:08.778848 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:08.778616    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5_odh-ai-gateway-infra(c3edd927-4b9b-4956-945c-057a96ba0229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" podUID="c3edd927-4b9b-4956-945c-057a96ba0229"
Jul 19 08:31:13.493851 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.493812    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:13.494357 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.494336    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:13.494435 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.494362    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef236c4-400b-4621-adef-7d0612005b45" containerName="revoke-keys"
Jul 19 08:31:13.498756 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.498736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:13.504055 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.504023    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:13.557162 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.557127    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh\" (UID: \"a9bf0e27-ff37-48e1-b024-32a3f3924090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:13.658521 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.658488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh\" (UID: \"a9bf0e27-ff37-48e1-b024-32a3f3924090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:13.659149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.659128    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh\" (UID: \"a9bf0e27-ff37-48e1-b024-32a3f3924090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:13.811667 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.811640    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:13.942211 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:13.942182    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:13.944144 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:13.944118    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bf0e27_ff37_48e1_b024_32a3f3924090.slice/crio-af466210c9403b023b7ab4bf45c3c18af14780d58b3ecd123d217824396ceb85 WatchSource:0}: Error finding container af466210c9403b023b7ab4bf45c3c18af14780d58b3ecd123d217824396ceb85: Status 404 returned error can't find the container with id af466210c9403b023b7ab4bf45c3c18af14780d58b3ecd123d217824396ceb85
Jul 19 08:31:14.819109 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:14.819010    2559 generic.go:356] "Generic (PLEG): container finished" podID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerID="ab161ca4cf1bed5d9f093a96e327936a3b1b273c99a497f78ad5e3831dbee018" exitCode=6
Jul 19 08:31:14.819109 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:14.819092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerDied","Data":"ab161ca4cf1bed5d9f093a96e327936a3b1b273c99a497f78ad5e3831dbee018"}
Jul 19 08:31:14.819620 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:14.819146    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerStarted","Data":"af466210c9403b023b7ab4bf45c3c18af14780d58b3ecd123d217824396ceb85"}
Jul 19 08:31:14.819620 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:14.819374    2559 scope.go:117] "RemoveContainer" containerID="ab161ca4cf1bed5d9f093a96e327936a3b1b273c99a497f78ad5e3831dbee018"
Jul 19 08:31:15.826371 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:15.826337    2559 generic.go:356] "Generic (PLEG): container finished" podID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerID="d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4" exitCode=6
Jul 19 08:31:15.826855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:15.826418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerDied","Data":"d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4"}
Jul 19 08:31:15.826855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:15.826534    2559 scope.go:117] "RemoveContainer" containerID="ab161ca4cf1bed5d9f093a96e327936a3b1b273c99a497f78ad5e3831dbee018"
Jul 19 08:31:15.826855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:15.826741    2559 scope.go:117] "RemoveContainer" containerID="d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4"
Jul 19 08:31:15.827018 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:15.826996    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh_odh-ai-gateway-infra(a9bf0e27-ff37-48e1-b024-32a3f3924090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090"
Jul 19 08:31:16.834162 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:16.834134    2559 scope.go:117] "RemoveContainer" containerID="d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4"
Jul 19 08:31:16.834540 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:16.834325    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh_odh-ai-gateway-infra(a9bf0e27-ff37-48e1-b024-32a3f3924090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090"
Jul 19 08:31:20.308797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.308759    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:20.313459 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.313440    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:20.318810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.318783    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:20.417831 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.417795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4\" (UID: \"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:20.518626 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.518591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4\" (UID: \"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:20.519213 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.519196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4\" (UID: \"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:20.625208 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.625129    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:20.753347 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.753316    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:20.754669 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:20.754640    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2ec9a2_de8b_4a74_ac6b_94510e5f53a2.slice/crio-9b931f04b369d8cc7b6673ead4468f2c275f95024fcadd3d477e2e8a330b58b6 WatchSource:0}: Error finding container 9b931f04b369d8cc7b6673ead4468f2c275f95024fcadd3d477e2e8a330b58b6: Status 404 returned error can't find the container with id 9b931f04b369d8cc7b6673ead4468f2c275f95024fcadd3d477e2e8a330b58b6
Jul 19 08:31:20.855994 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:20.855961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerStarted","Data":"9b931f04b369d8cc7b6673ead4468f2c275f95024fcadd3d477e2e8a330b58b6"}
Jul 19 08:31:21.862762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:21.862727    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerID="097780116f1b220425117467704700ee350e99410cdd94f022519bef7af1e310" exitCode=6
Jul 19 08:31:21.863146 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:21.862787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerDied","Data":"097780116f1b220425117467704700ee350e99410cdd94f022519bef7af1e310"}
Jul 19 08:31:21.863146 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:21.862978    2559 scope.go:117] "RemoveContainer" containerID="097780116f1b220425117467704700ee350e99410cdd94f022519bef7af1e310"
Jul 19 08:31:22.306028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.305987    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"]
Jul 19 08:31:22.309475 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.309458    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:22.315957 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.315929    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"]
Jul 19 08:31:22.436788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.436678    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2\" (UID: \"c001200b-3ee8-4644-812d-a86a5d8db0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:22.455026 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.455001    2559 scope.go:117] "RemoveContainer" containerID="9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca"
Jul 19 08:31:22.537444 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.537415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2\" (UID: \"c001200b-3ee8-4644-812d-a86a5d8db0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:22.538097 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.538076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2\" (UID: \"c001200b-3ee8-4644-812d-a86a5d8db0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:22.622723 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.622675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:22.757695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.757669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"]
Jul 19 08:31:22.760284 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:22.760253    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc001200b_3ee8_4644_812d_a86a5d8db0dc.slice/crio-f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f WatchSource:0}: Error finding container f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f: Status 404 returned error can't find the container with id f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f
Jul 19 08:31:22.870610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.870580    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerID="80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c" exitCode=6
Jul 19 08:31:22.871053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.870661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerDied","Data":"80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c"}
Jul 19 08:31:22.871053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.870748    2559 scope.go:117] "RemoveContainer" containerID="097780116f1b220425117467704700ee350e99410cdd94f022519bef7af1e310"
Jul 19 08:31:22.871053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.870955    2559 scope.go:117] "RemoveContainer" containerID="80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c"
Jul 19 08:31:22.871211 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:22.871180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4_odh-ai-gateway-infra(2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2"
Jul 19 08:31:22.872607 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.872583    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerStarted","Data":"00315c35f8f3b251b3f8bf8df3b89335d39c19b5bd1ca811ac3bd22dc91f2e5f"}
Jul 19 08:31:22.873827 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.873799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2" event={"ID":"c001200b-3ee8-4644-812d-a86a5d8db0dc","Type":"ContainerStarted","Data":"f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f"}
Jul 19 08:31:22.897477 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:22.897433    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" podStartSLOduration=17.897418456 podStartE2EDuration="17.897418456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:31:22.896639014 +0000 UTC m=+2919.406236913" watchObservedRunningTime="2026-07-19 08:31:22.897418456 +0000 UTC m=+2919.407016354"
Jul 19 08:31:23.880579 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.880548    2559 generic.go:356] "Generic (PLEG): container finished" podID="c001200b-3ee8-4644-812d-a86a5d8db0dc" containerID="cf19956fa56e7a0a416ab66c2abd5530be929db1a4d83e96673a87dc8a93c69f" exitCode=0
Jul 19 08:31:23.881041 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.880637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2" event={"ID":"c001200b-3ee8-4644-812d-a86a5d8db0dc","Type":"ContainerDied","Data":"cf19956fa56e7a0a416ab66c2abd5530be929db1a4d83e96673a87dc8a93c69f"}
Jul 19 08:31:23.882310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.882293    2559 scope.go:117] "RemoveContainer" containerID="80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c"
Jul 19 08:31:23.882545 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:23.882518    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4_odh-ai-gateway-infra(2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2"
Jul 19 08:31:23.883904 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.883884    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3edd927-4b9b-4956-945c-057a96ba0229" containerID="00315c35f8f3b251b3f8bf8df3b89335d39c19b5bd1ca811ac3bd22dc91f2e5f" exitCode=6
Jul 19 08:31:23.884008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.883928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerDied","Data":"00315c35f8f3b251b3f8bf8df3b89335d39c19b5bd1ca811ac3bd22dc91f2e5f"}
Jul 19 08:31:23.884008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.883965    2559 scope.go:117] "RemoveContainer" containerID="9cab958c66500cae6ff21e491ae50d816d892d0344e818f9b02704ec563be5ca"
Jul 19 08:31:23.884218 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.884194    2559 scope.go:117] "RemoveContainer" containerID="00315c35f8f3b251b3f8bf8df3b89335d39c19b5bd1ca811ac3bd22dc91f2e5f"
Jul 19 08:31:23.884383 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:23.884367    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5_odh-ai-gateway-infra(c3edd927-4b9b-4956-945c-057a96ba0229)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" podUID="c3edd927-4b9b-4956-945c-057a96ba0229"
Jul 19 08:31:23.905653 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:23.905625    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:25.070722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.070679    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:25.073275 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.073253    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:25.161989 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.161907    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca\") pod \"c3edd927-4b9b-4956-945c-057a96ba0229\" (UID: \"c3edd927-4b9b-4956-945c-057a96ba0229\") "
Jul 19 08:31:25.161989 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.161963    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca\") pod \"c001200b-3ee8-4644-812d-a86a5d8db0dc\" (UID: \"c001200b-3ee8-4644-812d-a86a5d8db0dc\") "
Jul 19 08:31:25.162281 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.162259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3edd927-4b9b-4956-945c-057a96ba0229" (UID: "c3edd927-4b9b-4956-945c-057a96ba0229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:25.162377 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.162356    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c001200b-3ee8-4644-812d-a86a5d8db0dc" (UID: "c001200b-3ee8-4644-812d-a86a5d8db0dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:25.263224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.263197    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3edd927-4b9b-4956-945c-057a96ba0229-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:25.263224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.263222    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c001200b-3ee8-4644-812d-a86a5d8db0dc-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:25.901422 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.901387    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"
Jul 19 08:31:25.901601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.901390    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5" event={"ID":"c3edd927-4b9b-4956-945c-057a96ba0229","Type":"ContainerDied","Data":"3cfa661ee092c63486c90c8aa025b33d1af539bbeec28348199ac59f013c536d"}
Jul 19 08:31:25.901601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.901519    2559 scope.go:117] "RemoveContainer" containerID="00315c35f8f3b251b3f8bf8df3b89335d39c19b5bd1ca811ac3bd22dc91f2e5f"
Jul 19 08:31:25.902971 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.902688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2" event={"ID":"c001200b-3ee8-4644-812d-a86a5d8db0dc","Type":"ContainerDied","Data":"f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f"}
Jul 19 08:31:25.902971 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.902754    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b4aad57f541bcf09d0df9e67ff7ce8ab67657d2a36611b61f193c0a297b14f"
Jul 19 08:31:25.902971 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.902720    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2"
Jul 19 08:31:25.932037 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.932013    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:25.934278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:25.934259    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ktkt5"]
Jul 19 08:31:26.460649 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:26.460615    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" path="/var/lib/kubelet/pods/c3edd927-4b9b-4956-945c-057a96ba0229/volumes"
Jul 19 08:31:27.553325 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553292    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553652    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553662    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553672    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553677    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553684    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c001200b-3ee8-4644-812d-a86a5d8db0dc" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c001200b-3ee8-4644-812d-a86a5d8db0dc" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553715    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553721    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553784    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553791    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553799    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3edd927-4b9b-4956-945c-057a96ba0229" containerName="revoke-keys"
Jul 19 08:31:27.553991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.553806    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c001200b-3ee8-4644-812d-a86a5d8db0dc" containerName="revoke-keys"
Jul 19 08:31:27.558140 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.558123    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:27.563272 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.563242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:27.687614 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.687582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2\" (UID: \"e4dcb204-2ea3-4055-b92b-834477991806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:27.788979 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.788941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2\" (UID: \"e4dcb204-2ea3-4055-b92b-834477991806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:27.789541 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.789514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2\" (UID: \"e4dcb204-2ea3-4055-b92b-834477991806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:27.870834 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:27.870804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:28.001935 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:28.001902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:28.003059 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:28.003024    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4dcb204_2ea3_4055_b92b_834477991806.slice/crio-80b470a6488ed34b670142028fefb0affa12dc9717d06420b61f384465f9c1e5 WatchSource:0}: Error finding container 80b470a6488ed34b670142028fefb0affa12dc9717d06420b61f384465f9c1e5: Status 404 returned error can't find the container with id 80b470a6488ed34b670142028fefb0affa12dc9717d06420b61f384465f9c1e5
Jul 19 08:31:28.921737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:28.921627    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4dcb204-2ea3-4055-b92b-834477991806" containerID="b9b7f7a7f6d5a6e0bd6c67df6ce74791f4f8c2a6ac5d558eb773dedc291f08e9" exitCode=6
Jul 19 08:31:28.921737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:28.921724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerDied","Data":"b9b7f7a7f6d5a6e0bd6c67df6ce74791f4f8c2a6ac5d558eb773dedc291f08e9"}
Jul 19 08:31:28.922141 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:28.921777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerStarted","Data":"80b470a6488ed34b670142028fefb0affa12dc9717d06420b61f384465f9c1e5"}
Jul 19 08:31:28.922141 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:28.921902    2559 scope.go:117] "RemoveContainer" containerID="b9b7f7a7f6d5a6e0bd6c67df6ce74791f4f8c2a6ac5d558eb773dedc291f08e9"
Jul 19 08:31:29.454829 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.454796    2559 scope.go:117] "RemoveContainer" containerID="d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4"
Jul 19 08:31:29.930164 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.930123    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerStarted","Data":"ef9f17a253adf1fbebaaa14123764577a4076974108c52d3f2821990a441d013"}
Jul 19 08:31:29.931961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.931932    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4dcb204-2ea3-4055-b92b-834477991806" containerID="d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2" exitCode=6
Jul 19 08:31:29.932099 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.931966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerDied","Data":"d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2"}
Jul 19 08:31:29.932099 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.932015    2559 scope.go:117] "RemoveContainer" containerID="b9b7f7a7f6d5a6e0bd6c67df6ce74791f4f8c2a6ac5d558eb773dedc291f08e9"
Jul 19 08:31:29.932239 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.932174    2559 scope.go:117] "RemoveContainer" containerID="d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2"
Jul 19 08:31:29.932438 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:29.932415    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2_odh-ai-gateway-infra(e4dcb204-2ea3-4055-b92b-834477991806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" podUID="e4dcb204-2ea3-4055-b92b-834477991806"
Jul 19 08:31:29.945815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:29.945756    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" podStartSLOduration=16.945738595 podStartE2EDuration="16.945738595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:31:29.943410409 +0000 UTC m=+2926.453008310" watchObservedRunningTime="2026-07-19 08:31:29.945738595 +0000 UTC m=+2926.455336498"
Jul 19 08:31:30.942046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.942014    2559 generic.go:356] "Generic (PLEG): container finished" podID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerID="ef9f17a253adf1fbebaaa14123764577a4076974108c52d3f2821990a441d013" exitCode=6
Jul 19 08:31:30.942472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.942095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerDied","Data":"ef9f17a253adf1fbebaaa14123764577a4076974108c52d3f2821990a441d013"}
Jul 19 08:31:30.942472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.942164    2559 scope.go:117] "RemoveContainer" containerID="d3460615e0edec938daa4669f1e9eeb6269a817285324e4c03041c3437329ad4"
Jul 19 08:31:30.942611 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.942494    2559 scope.go:117] "RemoveContainer" containerID="ef9f17a253adf1fbebaaa14123764577a4076974108c52d3f2821990a441d013"
Jul 19 08:31:30.942786 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:30.942762    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh_odh-ai-gateway-infra(a9bf0e27-ff37-48e1-b024-32a3f3924090)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090"
Jul 19 08:31:30.943932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.943917    2559 scope.go:117] "RemoveContainer" containerID="d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2"
Jul 19 08:31:30.944139 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:30.944112    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2_odh-ai-gateway-infra(e4dcb204-2ea3-4055-b92b-834477991806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" podUID="e4dcb204-2ea3-4055-b92b-834477991806"
Jul 19 08:31:30.953279 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:30.953252    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:32.081675 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.081649    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:32.227825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.227730    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca\") pod \"a9bf0e27-ff37-48e1-b024-32a3f3924090\" (UID: \"a9bf0e27-ff37-48e1-b024-32a3f3924090\") "
Jul 19 08:31:32.228061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.228039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9bf0e27-ff37-48e1-b024-32a3f3924090" (UID: "a9bf0e27-ff37-48e1-b024-32a3f3924090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:32.328944 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.328913    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9bf0e27-ff37-48e1-b024-32a3f3924090-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:32.957344 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.957315    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"
Jul 19 08:31:32.957344 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.957336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh" event={"ID":"a9bf0e27-ff37-48e1-b024-32a3f3924090","Type":"ContainerDied","Data":"af466210c9403b023b7ab4bf45c3c18af14780d58b3ecd123d217824396ceb85"}
Jul 19 08:31:32.957576 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.957389    2559 scope.go:117] "RemoveContainer" containerID="ef9f17a253adf1fbebaaa14123764577a4076974108c52d3f2821990a441d013"
Jul 19 08:31:32.973035 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.973005    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:32.978828 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:32.978805    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6cglh"]
Jul 19 08:31:34.461225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:34.461188    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" path="/var/lib/kubelet/pods/a9bf0e27-ff37-48e1-b024-32a3f3924090/volumes"
Jul 19 08:31:35.455407 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:35.455371    2559 scope.go:117] "RemoveContainer" containerID="80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c"
Jul 19 08:31:35.975871 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:35.975836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerStarted","Data":"59558b67c479a496402a0aff51dc99c35305c126b1a08d060978425de853249c"}
Jul 19 08:31:35.991486 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:35.991434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" podStartSLOduration=15.99141956 podStartE2EDuration="15.99141956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:31:35.99003567 +0000 UTC m=+2932.499633570" watchObservedRunningTime="2026-07-19 08:31:35.99141956 +0000 UTC m=+2932.501017458"
Jul 19 08:31:36.983161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:36.983128    2559 generic.go:356] "Generic (PLEG): container finished" podID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerID="59558b67c479a496402a0aff51dc99c35305c126b1a08d060978425de853249c" exitCode=6
Jul 19 08:31:36.983637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:36.983201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerDied","Data":"59558b67c479a496402a0aff51dc99c35305c126b1a08d060978425de853249c"}
Jul 19 08:31:36.983637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:36.983260    2559 scope.go:117] "RemoveContainer" containerID="80c10ef0ef7db814283a821222ae5015690c0be45fe4b243a9df1eedfbfbdc8c"
Jul 19 08:31:36.983637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:36.983525    2559 scope.go:117] "RemoveContainer" containerID="59558b67c479a496402a0aff51dc99c35305c126b1a08d060978425de853249c"
Jul 19 08:31:36.983809 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:36.983749    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4_odh-ai-gateway-infra(2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2"
Jul 19 08:31:37.006323 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:37.006298    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:38.123609 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:38.123586    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:38.285912 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:38.285830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca\") pod \"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2\" (UID: \"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2\") "
Jul 19 08:31:38.286178 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:38.286156    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" (UID: "2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:38.386765 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:38.386732    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:39.002523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:39.002493    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"
Jul 19 08:31:39.002741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:39.002496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4" event={"ID":"2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2","Type":"ContainerDied","Data":"9b931f04b369d8cc7b6673ead4468f2c275f95024fcadd3d477e2e8a330b58b6"}
Jul 19 08:31:39.002741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:39.002649    2559 scope.go:117] "RemoveContainer" containerID="59558b67c479a496402a0aff51dc99c35305c126b1a08d060978425de853249c"
Jul 19 08:31:39.024219 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:39.024192    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:39.030963 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:39.030940    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fftp4"]
Jul 19 08:31:40.461662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:40.461627    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" path="/var/lib/kubelet/pods/2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2/volumes"
Jul 19 08:31:44.462318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:44.462283    2559 scope.go:117] "RemoveContainer" containerID="d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2"
Jul 19 08:31:45.038478 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:45.038393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerStarted","Data":"cad83379fb845a828557841c525dc87718f93156d786a85d5817470e62a37949"}
Jul 19 08:31:45.053835 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:45.053787    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" podStartSLOduration=18.053770185 podStartE2EDuration="18.053770185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:31:45.052394013 +0000 UTC m=+2941.561991912" watchObservedRunningTime="2026-07-19 08:31:45.053770185 +0000 UTC m=+2941.563368083"
Jul 19 08:31:46.045507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:46.045473    2559 generic.go:356] "Generic (PLEG): container finished" podID="e4dcb204-2ea3-4055-b92b-834477991806" containerID="cad83379fb845a828557841c525dc87718f93156d786a85d5817470e62a37949" exitCode=6
Jul 19 08:31:46.045947 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:46.045549    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerDied","Data":"cad83379fb845a828557841c525dc87718f93156d786a85d5817470e62a37949"}
Jul 19 08:31:46.045947 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:46.045625    2559 scope.go:117] "RemoveContainer" containerID="d8bce32bd1410b4b615211d3fe105d0b06955cc7597dd0bef61a13be3d02b2a2"
Jul 19 08:31:46.046053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:46.045973    2559 scope.go:117] "RemoveContainer" containerID="cad83379fb845a828557841c525dc87718f93156d786a85d5817470e62a37949"
Jul 19 08:31:46.046223 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:31:46.046200    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2_odh-ai-gateway-infra(e4dcb204-2ea3-4055-b92b-834477991806)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" podUID="e4dcb204-2ea3-4055-b92b-834477991806"
Jul 19 08:31:46.061975 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:46.061951    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:47.186948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:47.186922    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:47.367530 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:47.367490    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca\") pod \"e4dcb204-2ea3-4055-b92b-834477991806\" (UID: \"e4dcb204-2ea3-4055-b92b-834477991806\") "
Jul 19 08:31:47.367867 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:47.367845    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e4dcb204-2ea3-4055-b92b-834477991806" (UID: "e4dcb204-2ea3-4055-b92b-834477991806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:47.468094 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:47.468062    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4dcb204-2ea3-4055-b92b-834477991806-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:48.061109 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.061078    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"
Jul 19 08:31:48.061109 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.061097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2" event={"ID":"e4dcb204-2ea3-4055-b92b-834477991806","Type":"ContainerDied","Data":"80b470a6488ed34b670142028fefb0affa12dc9717d06420b61f384465f9c1e5"}
Jul 19 08:31:48.061345 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.061162    2559 scope.go:117] "RemoveContainer" containerID="cad83379fb845a828557841c525dc87718f93156d786a85d5817470e62a37949"
Jul 19 08:31:48.089955 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.089927    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:48.097901 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.097877    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-s7gb2"]
Jul 19 08:31:48.460866 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.460789    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4dcb204-2ea3-4055-b92b-834477991806" path="/var/lib/kubelet/pods/e4dcb204-2ea3-4055-b92b-834477991806/volumes"
Jul 19 08:31:48.461216 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461067    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"]
Jul 19 08:31:48.461373 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461361    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461375    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461387    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461400    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461405    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461410 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461412    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461417    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461425    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461430    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461442    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461447    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461452    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461457    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461511    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461518    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461524    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461532    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461538    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461542    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9bf0e27-ff37-48e1-b024-32a3f3924090" containerName="revoke-keys"
Jul 19 08:31:48.461590 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.461548    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:31:48.465932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.465905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:48.468432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.468410    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"]
Jul 19 08:31:48.468942 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.468923    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:31:48.469259 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.469228    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:31:48.579149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.579118    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg\" (UID: \"ea10278a-a25b-4ed0-bdf0-a69d58de7838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:48.680937 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.680905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg\" (UID: \"ea10278a-a25b-4ed0-bdf0-a69d58de7838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:48.681563 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.681535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg\" (UID: \"ea10278a-a25b-4ed0-bdf0-a69d58de7838\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:48.778683 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.778602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:48.911114 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:48.911091    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"]
Jul 19 08:31:48.912783 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:31:48.912751    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea10278a_a25b_4ed0_bdf0_a69d58de7838.slice/crio-a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b WatchSource:0}: Error finding container a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b: Status 404 returned error can't find the container with id a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b
Jul 19 08:31:49.074305 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:49.074270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg" event={"ID":"ea10278a-a25b-4ed0-bdf0-a69d58de7838","Type":"ContainerStarted","Data":"a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b"}
Jul 19 08:31:50.081917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:50.081882    2559 generic.go:356] "Generic (PLEG): container finished" podID="ea10278a-a25b-4ed0-bdf0-a69d58de7838" containerID="90dc54a6d6d843b06bd19a2c6674b6cf279910ed0ff9ae3c4e3b875f4edb777e" exitCode=0
Jul 19 08:31:50.082305 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:50.081956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg" event={"ID":"ea10278a-a25b-4ed0-bdf0-a69d58de7838","Type":"ContainerDied","Data":"90dc54a6d6d843b06bd19a2c6674b6cf279910ed0ff9ae3c4e3b875f4edb777e"}
Jul 19 08:31:51.219034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:51.219011    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:51.303006 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:51.302977    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca\") pod \"ea10278a-a25b-4ed0-bdf0-a69d58de7838\" (UID: \"ea10278a-a25b-4ed0-bdf0-a69d58de7838\") "
Jul 19 08:31:51.303352 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:51.303326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea10278a-a25b-4ed0-bdf0-a69d58de7838" (UID: "ea10278a-a25b-4ed0-bdf0-a69d58de7838"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:31:51.403930 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:51.403850    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea10278a-a25b-4ed0-bdf0-a69d58de7838-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:31:52.095067 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:52.095032    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg"
Jul 19 08:31:52.095237 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:52.095065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg" event={"ID":"ea10278a-a25b-4ed0-bdf0-a69d58de7838","Type":"ContainerDied","Data":"a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b"}
Jul 19 08:31:52.095237 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:31:52.095113    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40d36503d919aba1462ebdd8a1bc40efbc804ce700942f7e70eba5e4cde122b"
Jul 19 08:32:03.550314 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550278    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550646    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550657    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550667    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550672    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550683    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea10278a-a25b-4ed0-bdf0-a69d58de7838" containerName="revoke-keys"
Jul 19 08:32:03.550737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550689    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea10278a-a25b-4ed0-bdf0-a69d58de7838" containerName="revoke-keys"
Jul 19 08:32:03.550948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550757    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea10278a-a25b-4ed0-bdf0-a69d58de7838" containerName="revoke-keys"
Jul 19 08:32:03.550948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550767    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e2ec9a2-de8b-4a74-ac6b-94510e5f53a2" containerName="revoke-keys"
Jul 19 08:32:03.550948 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.550774    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4dcb204-2ea3-4055-b92b-834477991806" containerName="revoke-keys"
Jul 19 08:32:03.553837 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.553814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:03.556431 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.556408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:32:03.556741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.556717    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:32:03.561593 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.561569    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:03.607637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.607600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4\" (UID: \"8447c8aa-7a32-483c-868a-24036941632b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:03.709080 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.709049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4\" (UID: \"8447c8aa-7a32-483c-868a-24036941632b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:03.709612 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.709594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4\" (UID: \"8447c8aa-7a32-483c-868a-24036941632b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:03.866112 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.866074    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:03.993127 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:03.993107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:03.994992 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:32:03.994964    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8447c8aa_7a32_483c_868a_24036941632b.slice/crio-b43f00a3cd9a5ad4c3e9a7ab20eb7c806b5421aabc86f8d42c25fd4930089717 WatchSource:0}: Error finding container b43f00a3cd9a5ad4c3e9a7ab20eb7c806b5421aabc86f8d42c25fd4930089717: Status 404 returned error can't find the container with id b43f00a3cd9a5ad4c3e9a7ab20eb7c806b5421aabc86f8d42c25fd4930089717
Jul 19 08:32:04.163918 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:04.163844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" event={"ID":"8447c8aa-7a32-483c-868a-24036941632b","Type":"ContainerStarted","Data":"b43f00a3cd9a5ad4c3e9a7ab20eb7c806b5421aabc86f8d42c25fd4930089717"}
Jul 19 08:32:05.170600 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:05.170562    2559 generic.go:356] "Generic (PLEG): container finished" podID="8447c8aa-7a32-483c-868a-24036941632b" containerID="e2b9406f3fc5cda1220c8f3ac65d278d63c57d2ea4e769a64af7b84287356282" exitCode=6
Jul 19 08:32:05.171053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:05.170631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" event={"ID":"8447c8aa-7a32-483c-868a-24036941632b","Type":"ContainerDied","Data":"e2b9406f3fc5cda1220c8f3ac65d278d63c57d2ea4e769a64af7b84287356282"}
Jul 19 08:32:05.171053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:05.170838    2559 scope.go:117] "RemoveContainer" containerID="e2b9406f3fc5cda1220c8f3ac65d278d63c57d2ea4e769a64af7b84287356282"
Jul 19 08:32:06.177534 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:06.177500    2559 generic.go:356] "Generic (PLEG): container finished" podID="8447c8aa-7a32-483c-868a-24036941632b" containerID="21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754" exitCode=6
Jul 19 08:32:06.177986 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:06.177547    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" event={"ID":"8447c8aa-7a32-483c-868a-24036941632b","Type":"ContainerDied","Data":"21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754"}
Jul 19 08:32:06.177986 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:06.177623    2559 scope.go:117] "RemoveContainer" containerID="e2b9406f3fc5cda1220c8f3ac65d278d63c57d2ea4e769a64af7b84287356282"
Jul 19 08:32:06.177986 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:06.177851    2559 scope.go:117] "RemoveContainer" containerID="21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754"
Jul 19 08:32:06.178131 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:32:06.178090    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4_odh-ai-gateway-infra(8447c8aa-7a32-483c-868a-24036941632b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" podUID="8447c8aa-7a32-483c-868a-24036941632b"
Jul 19 08:32:07.184116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:07.184086    2559 scope.go:117] "RemoveContainer" containerID="21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754"
Jul 19 08:32:07.184612 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:32:07.184297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4_odh-ai-gateway-infra(8447c8aa-7a32-483c-868a-24036941632b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" podUID="8447c8aa-7a32-483c-868a-24036941632b"
Jul 19 08:32:20.461265 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:20.461235    2559 scope.go:117] "RemoveContainer" containerID="21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754"
Jul 19 08:32:21.273353 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:21.273320    2559 generic.go:356] "Generic (PLEG): container finished" podID="8447c8aa-7a32-483c-868a-24036941632b" containerID="364d0a4334deeeb9109239926452720673bec78a91494a0dd72ffb78c89a54a0" exitCode=6
Jul 19 08:32:21.273527 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:21.273387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" event={"ID":"8447c8aa-7a32-483c-868a-24036941632b","Type":"ContainerDied","Data":"364d0a4334deeeb9109239926452720673bec78a91494a0dd72ffb78c89a54a0"}
Jul 19 08:32:21.273527 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:21.273445    2559 scope.go:117] "RemoveContainer" containerID="21472cd43f6a2c204dcd31712e1249f923918c0a3d11994dfcfab0df78486754"
Jul 19 08:32:21.273712 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:21.273684    2559 scope.go:117] "RemoveContainer" containerID="364d0a4334deeeb9109239926452720673bec78a91494a0dd72ffb78c89a54a0"
Jul 19 08:32:21.273934 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:32:21.273914    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4_odh-ai-gateway-infra(8447c8aa-7a32-483c-868a-24036941632b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" podUID="8447c8aa-7a32-483c-868a-24036941632b"
Jul 19 08:32:22.298192 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:22.298159    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:22.436489 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:22.436466    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:22.576762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:22.576735    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca\") pod \"8447c8aa-7a32-483c-868a-24036941632b\" (UID: \"8447c8aa-7a32-483c-868a-24036941632b\") "
Jul 19 08:32:22.577077 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:22.577053    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8447c8aa-7a32-483c-868a-24036941632b" (UID: "8447c8aa-7a32-483c-868a-24036941632b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:32:22.677569 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:22.677541    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8447c8aa-7a32-483c-868a-24036941632b-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:32:23.289384 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:23.289223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4" event={"ID":"8447c8aa-7a32-483c-868a-24036941632b","Type":"ContainerDied","Data":"b43f00a3cd9a5ad4c3e9a7ab20eb7c806b5421aabc86f8d42c25fd4930089717"}
Jul 19 08:32:23.289384 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:23.289300    2559 scope.go:117] "RemoveContainer" containerID="364d0a4334deeeb9109239926452720673bec78a91494a0dd72ffb78c89a54a0"
Jul 19 08:32:23.289738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:23.289688    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"
Jul 19 08:32:23.315601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:23.315574    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:23.326528 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:23.326487    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-mfxn4"]
Jul 19 08:32:24.460784 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:24.460752    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8447c8aa-7a32-483c-868a-24036941632b" path="/var/lib/kubelet/pods/8447c8aa-7a32-483c-868a-24036941632b/volumes"
Jul 19 08:32:38.505338 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505302    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"]
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505659    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505670    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505688    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505693    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505715    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505721    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505778    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.505855 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.505785    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:38.510085 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.510068    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:38.512690 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.512666    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:32:38.512965 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.512666    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:32:38.515154 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.515131    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"]
Jul 19 08:32:38.623174 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.623142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n\" (UID: \"c292ffa7-b861-461b-8fb8-ffabc2907004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:38.723869 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.723826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n\" (UID: \"c292ffa7-b861-461b-8fb8-ffabc2907004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:38.724640 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.724614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n\" (UID: \"c292ffa7-b861-461b-8fb8-ffabc2907004\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:38.823103 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.823070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:38.969229 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:38.969200    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"]
Jul 19 08:32:38.970203 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:32:38.970177    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc292ffa7_b861_461b_8fb8_ffabc2907004.slice/crio-87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718 WatchSource:0}: Error finding container 87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718: Status 404 returned error can't find the container with id 87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718
Jul 19 08:32:39.379752 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:39.379718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n" event={"ID":"c292ffa7-b861-461b-8fb8-ffabc2907004","Type":"ContainerStarted","Data":"9d7d06348c723ccf4f6b0739a245d8902226bb00de6f3f75e28d5fca33e90b4f"}
Jul 19 08:32:39.379927 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:39.379772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n" event={"ID":"c292ffa7-b861-461b-8fb8-ffabc2907004","Type":"ContainerStarted","Data":"87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718"}
Jul 19 08:32:39.394561 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:39.394506    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n" podStartSLOduration=1.394489887 podStartE2EDuration="1.394489887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:39.393622103 +0000 UTC m=+2995.903220003" watchObservedRunningTime="2026-07-19 08:32:39.394489887 +0000 UTC m=+2995.904087807"
Jul 19 08:32:40.386762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:40.386728    2559 generic.go:356] "Generic (PLEG): container finished" podID="c292ffa7-b861-461b-8fb8-ffabc2907004" containerID="9d7d06348c723ccf4f6b0739a245d8902226bb00de6f3f75e28d5fca33e90b4f" exitCode=0
Jul 19 08:32:40.387171 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:40.386836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n" event={"ID":"c292ffa7-b861-461b-8fb8-ffabc2907004","Type":"ContainerDied","Data":"9d7d06348c723ccf4f6b0739a245d8902226bb00de6f3f75e28d5fca33e90b4f"}
Jul 19 08:32:41.524572 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:41.524543    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:41.651969 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:41.651868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca\") pod \"c292ffa7-b861-461b-8fb8-ffabc2907004\" (UID: \"c292ffa7-b861-461b-8fb8-ffabc2907004\") "
Jul 19 08:32:41.652226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:41.652201    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c292ffa7-b861-461b-8fb8-ffabc2907004" (UID: "c292ffa7-b861-461b-8fb8-ffabc2907004"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:32:41.753271 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:41.753237    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c292ffa7-b861-461b-8fb8-ffabc2907004-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:32:42.399862 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:42.399836    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n"
Jul 19 08:32:42.400023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:42.399861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n" event={"ID":"c292ffa7-b861-461b-8fb8-ffabc2907004","Type":"ContainerDied","Data":"87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718"}
Jul 19 08:32:42.400023 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:42.399902    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a7e5ccb931a983353dea44b8b44420c6070376cbc478586dbd444bbe2df718"
Jul 19 08:32:53.625330 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.625291    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:32:53.625746 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.625652    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c292ffa7-b861-461b-8fb8-ffabc2907004" containerName="revoke-keys"
Jul 19 08:32:53.625746 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.625664    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c292ffa7-b861-461b-8fb8-ffabc2907004" containerName="revoke-keys"
Jul 19 08:32:53.625837 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.625754    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c292ffa7-b861-461b-8fb8-ffabc2907004" containerName="revoke-keys"
Jul 19 08:32:53.625837 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.625770    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8447c8aa-7a32-483c-868a-24036941632b" containerName="revoke-keys"
Jul 19 08:32:53.628564 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.628548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:32:53.631199 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.631158    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:32:53.634724 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.632386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:32:53.639655 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.639622    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:32:53.656620 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.656592    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7\" (UID: \"f557a895-6172-42a7-b17d-acbd82058cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:32:53.757734 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.757687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7\" (UID: \"f557a895-6172-42a7-b17d-acbd82058cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:32:53.758525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.758501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7\" (UID: \"f557a895-6172-42a7-b17d-acbd82058cd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:32:53.948406 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:53.948306    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:32:54.081002 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:54.080938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:32:54.083614 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:32:54.083583    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf557a895_6172_42a7_b17d_acbd82058cd1.slice/crio-f00204967cbbcfe0d7d34fa3a9c40cc512bcd8296e9c183459c004d69eef9705 WatchSource:0}: Error finding container f00204967cbbcfe0d7d34fa3a9c40cc512bcd8296e9c183459c004d69eef9705: Status 404 returned error can't find the container with id f00204967cbbcfe0d7d34fa3a9c40cc512bcd8296e9c183459c004d69eef9705
Jul 19 08:32:54.466001 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:54.465961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerStarted","Data":"fdfa74b7d3a2d124d26e5cefc3cb7c94a400ef13d042f87eb61f6ec3d05bb062"}
Jul 19 08:32:54.466189 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:54.466015    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerStarted","Data":"f00204967cbbcfe0d7d34fa3a9c40cc512bcd8296e9c183459c004d69eef9705"}
Jul 19 08:32:54.480225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:54.480182    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" podStartSLOduration=1.480168657 podStartE2EDuration="1.480168657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:32:54.480060498 +0000 UTC m=+3010.989658410" watchObservedRunningTime="2026-07-19 08:32:54.480168657 +0000 UTC m=+3010.989766567"
Jul 19 08:32:55.472662 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:55.472624    2559 generic.go:356] "Generic (PLEG): container finished" podID="f557a895-6172-42a7-b17d-acbd82058cd1" containerID="fdfa74b7d3a2d124d26e5cefc3cb7c94a400ef13d042f87eb61f6ec3d05bb062" exitCode=6
Jul 19 08:32:55.473034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:55.472721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerDied","Data":"fdfa74b7d3a2d124d26e5cefc3cb7c94a400ef13d042f87eb61f6ec3d05bb062"}
Jul 19 08:32:55.473034 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:55.472922    2559 scope.go:117] "RemoveContainer" containerID="fdfa74b7d3a2d124d26e5cefc3cb7c94a400ef13d042f87eb61f6ec3d05bb062"
Jul 19 08:32:56.480087 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:56.479997    2559 generic.go:356] "Generic (PLEG): container finished" podID="f557a895-6172-42a7-b17d-acbd82058cd1" containerID="8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b" exitCode=6
Jul 19 08:32:56.480537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:56.480085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerDied","Data":"8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b"}
Jul 19 08:32:56.480537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:56.480140    2559 scope.go:117] "RemoveContainer" containerID="fdfa74b7d3a2d124d26e5cefc3cb7c94a400ef13d042f87eb61f6ec3d05bb062"
Jul 19 08:32:56.480537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:56.480340    2559 scope.go:117] "RemoveContainer" containerID="8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b"
Jul 19 08:32:56.480670 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:32:56.480561    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7_odh-ai-gateway-infra(f557a895-6172-42a7-b17d-acbd82058cd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" podUID="f557a895-6172-42a7-b17d-acbd82058cd1"
Jul 19 08:32:57.487219 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:32:57.487193    2559 scope.go:117] "RemoveContainer" containerID="8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b"
Jul 19 08:32:57.487605 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:32:57.487415    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7_odh-ai-gateway-infra(f557a895-6172-42a7-b17d-acbd82058cd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" podUID="f557a895-6172-42a7-b17d-acbd82058cd1"
Jul 19 08:33:12.455097 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:12.455059    2559 scope.go:117] "RemoveContainer" containerID="8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b"
Jul 19 08:33:13.574891 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:13.574863    2559 generic.go:356] "Generic (PLEG): container finished" podID="f557a895-6172-42a7-b17d-acbd82058cd1" containerID="062f3e690da5a4f70ddafa877591ef04463ffb795d7de30aaff3db68dd575e70" exitCode=6
Jul 19 08:33:13.575317 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:13.574938    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerDied","Data":"062f3e690da5a4f70ddafa877591ef04463ffb795d7de30aaff3db68dd575e70"}
Jul 19 08:33:13.575317 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:13.575000    2559 scope.go:117] "RemoveContainer" containerID="8a8c4eaedf42abfa60be2c37aa8890ddedfb367f9038f0533c26d8465de2d61b"
Jul 19 08:33:13.575317 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:13.575298    2559 scope.go:117] "RemoveContainer" containerID="062f3e690da5a4f70ddafa877591ef04463ffb795d7de30aaff3db68dd575e70"
Jul 19 08:33:13.575507 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:33:13.575491    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7_odh-ai-gateway-infra(f557a895-6172-42a7-b17d-acbd82058cd1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" podUID="f557a895-6172-42a7-b17d-acbd82058cd1"
Jul 19 08:33:14.599875 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:14.599840    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:33:14.732084 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:14.732060    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:33:14.849133 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:14.849107    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca\") pod \"f557a895-6172-42a7-b17d-acbd82058cd1\" (UID: \"f557a895-6172-42a7-b17d-acbd82058cd1\") "
Jul 19 08:33:14.849460 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:14.849439    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f557a895-6172-42a7-b17d-acbd82058cd1" (UID: "f557a895-6172-42a7-b17d-acbd82058cd1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:33:14.949782 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:14.949686    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f557a895-6172-42a7-b17d-acbd82058cd1-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:33:15.588228 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:15.588200    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"
Jul 19 08:33:15.588434 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:15.588239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7" event={"ID":"f557a895-6172-42a7-b17d-acbd82058cd1","Type":"ContainerDied","Data":"f00204967cbbcfe0d7d34fa3a9c40cc512bcd8296e9c183459c004d69eef9705"}
Jul 19 08:33:15.588434 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:15.588299    2559 scope.go:117] "RemoveContainer" containerID="062f3e690da5a4f70ddafa877591ef04463ffb795d7de30aaff3db68dd575e70"
Jul 19 08:33:15.611082 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:15.611054    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:33:15.617952 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:15.617929    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-d64x7"]
Jul 19 08:33:16.461165 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:16.461130    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" path="/var/lib/kubelet/pods/f557a895-6172-42a7-b17d-acbd82058cd1/volumes"
Jul 19 08:33:50.437385 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437348    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:33:50.437825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437731    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.437825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437743    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.437825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437760    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.437825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437766    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.437825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437826    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.437996 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.437834    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:33:50.442031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.442014    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:33:50.444918 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.444897    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:33:50.445066 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.444997    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:33:50.447627 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.447584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:33:50.573167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.573132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh\" (UID: \"149d0e64-fc0d-4bb1-af7a-70611a371e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:33:50.673889 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.673856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh\" (UID: \"149d0e64-fc0d-4bb1-af7a-70611a371e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:33:50.674513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.674487    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh\" (UID: \"149d0e64-fc0d-4bb1-af7a-70611a371e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:33:50.754587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.754504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:33:50.884431 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:50.884395    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:33:50.887363 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:33:50.887337    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149d0e64_fc0d_4bb1_af7a_70611a371e41.slice/crio-c25271a5e860026f3f151a08a9da83f87da0eb5e908631fd39739955c003f17a WatchSource:0}: Error finding container c25271a5e860026f3f151a08a9da83f87da0eb5e908631fd39739955c003f17a: Status 404 returned error can't find the container with id c25271a5e860026f3f151a08a9da83f87da0eb5e908631fd39739955c003f17a
Jul 19 08:33:51.791920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:51.791822    2559 generic.go:356] "Generic (PLEG): container finished" podID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerID="49f8a1b5e839a136cd67d1c95b564cd5b322bd35ec6c188a0cebb27c531143e1" exitCode=6
Jul 19 08:33:51.791920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:51.791873    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerDied","Data":"49f8a1b5e839a136cd67d1c95b564cd5b322bd35ec6c188a0cebb27c531143e1"}
Jul 19 08:33:51.791920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:51.791916    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerStarted","Data":"c25271a5e860026f3f151a08a9da83f87da0eb5e908631fd39739955c003f17a"}
Jul 19 08:33:51.792471 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:51.792133    2559 scope.go:117] "RemoveContainer" containerID="49f8a1b5e839a136cd67d1c95b564cd5b322bd35ec6c188a0cebb27c531143e1"
Jul 19 08:33:52.799307 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:52.799211    2559 generic.go:356] "Generic (PLEG): container finished" podID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerID="f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f" exitCode=6
Jul 19 08:33:52.799307 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:52.799288    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerDied","Data":"f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f"}
Jul 19 08:33:52.799825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:52.799360    2559 scope.go:117] "RemoveContainer" containerID="49f8a1b5e839a136cd67d1c95b564cd5b322bd35ec6c188a0cebb27c531143e1"
Jul 19 08:33:52.799825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:52.799482    2559 scope.go:117] "RemoveContainer" containerID="f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f"
Jul 19 08:33:52.799825 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:33:52.799740    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh_odh-ai-gateway-infra(149d0e64-fc0d-4bb1-af7a-70611a371e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41"
Jul 19 08:33:53.806234 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:33:53.806208    2559 scope.go:117] "RemoveContainer" containerID="f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f"
Jul 19 08:33:53.806626 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:33:53.806395    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh_odh-ai-gateway-infra(149d0e64-fc0d-4bb1-af7a-70611a371e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41"
Jul 19 08:34:05.374801 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.374760    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:05.375186 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.375159    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:34:05.375186 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.375172    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:34:05.375291 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.375242    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f557a895-6172-42a7-b17d-acbd82058cd1" containerName="revoke-keys"
Jul 19 08:34:05.378071 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.378056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:05.386937 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.386911    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:05.512458 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.512420    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l\" (UID: \"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:05.613784 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.613748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l\" (UID: \"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:05.614414 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.614388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l\" (UID: \"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:05.689301 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.689219    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:05.818545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.818515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:05.820859 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:34:05.820832    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f6bc16_f0a3_40d9_8f64_9cbd7b8362f8.slice/crio-0e4441e464011272872a5603ba33a5ca4d526b7487514dfc80b12df14f4cfe95 WatchSource:0}: Error finding container 0e4441e464011272872a5603ba33a5ca4d526b7487514dfc80b12df14f4cfe95: Status 404 returned error can't find the container with id 0e4441e464011272872a5603ba33a5ca4d526b7487514dfc80b12df14f4cfe95
Jul 19 08:34:05.873810 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:05.873774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerStarted","Data":"0e4441e464011272872a5603ba33a5ca4d526b7487514dfc80b12df14f4cfe95"}
Jul 19 08:34:06.455402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.455316    2559 scope.go:117] "RemoveContainer" containerID="f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f"
Jul 19 08:34:06.882281 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.882247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerStarted","Data":"4cf6acac509c31a781a97dcafa148bdbfd7c2f6d05bf4ade980cd9ac04c4e1a9"}
Jul 19 08:34:06.883904 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.883879    2559 generic.go:356] "Generic (PLEG): container finished" podID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerID="98f636d6431b8298dcff5cf81aa90c969cfa599d30264678e60a7743fb5e1b04" exitCode=6
Jul 19 08:34:06.884035 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.883925    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerDied","Data":"98f636d6431b8298dcff5cf81aa90c969cfa599d30264678e60a7743fb5e1b04"}
Jul 19 08:34:06.884134 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.884122    2559 scope.go:117] "RemoveContainer" containerID="98f636d6431b8298dcff5cf81aa90c969cfa599d30264678e60a7743fb5e1b04"
Jul 19 08:34:06.900476 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:06.900434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" podStartSLOduration=16.900419918 podStartE2EDuration="16.900419918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:06.897379679 +0000 UTC m=+3083.406977578" watchObservedRunningTime="2026-07-19 08:34:06.900419918 +0000 UTC m=+3083.410017816"
Jul 19 08:34:07.890750 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.890695    2559 generic.go:356] "Generic (PLEG): container finished" podID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerID="4cf6acac509c31a781a97dcafa148bdbfd7c2f6d05bf4ade980cd9ac04c4e1a9" exitCode=6
Jul 19 08:34:07.891225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.890773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerDied","Data":"4cf6acac509c31a781a97dcafa148bdbfd7c2f6d05bf4ade980cd9ac04c4e1a9"}
Jul 19 08:34:07.891225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.890845    2559 scope.go:117] "RemoveContainer" containerID="f282657f455bf76acbb2332969dd9b051a45138a978cd0ff67c172d638ea415f"
Jul 19 08:34:07.891361 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.891240    2559 scope.go:117] "RemoveContainer" containerID="4cf6acac509c31a781a97dcafa148bdbfd7c2f6d05bf4ade980cd9ac04c4e1a9"
Jul 19 08:34:07.891535 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:07.891508    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh_odh-ai-gateway-infra(149d0e64-fc0d-4bb1-af7a-70611a371e41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41"
Jul 19 08:34:07.892775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.892746    2559 generic.go:356] "Generic (PLEG): container finished" podID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerID="38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de" exitCode=6
Jul 19 08:34:07.892913 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.892827    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerDied","Data":"38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de"}
Jul 19 08:34:07.892993 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.892980    2559 scope.go:117] "RemoveContainer" containerID="38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de"
Jul 19 08:34:07.893149 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:07.893134    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l_odh-ai-gateway-infra(38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8"
Jul 19 08:34:07.901965 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.901951    2559 scope.go:117] "RemoveContainer" containerID="98f636d6431b8298dcff5cf81aa90c969cfa599d30264678e60a7743fb5e1b04"
Jul 19 08:34:07.907434 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:07.907415    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:34:08.901611 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:08.901590    2559 scope.go:117] "RemoveContainer" containerID="38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de"
Jul 19 08:34:08.901942 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:08.901810    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l_odh-ai-gateway-infra(38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8"
Jul 19 08:34:09.034638 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.034614    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:34:09.144524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.144491    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca\") pod \"149d0e64-fc0d-4bb1-af7a-70611a371e41\" (UID: \"149d0e64-fc0d-4bb1-af7a-70611a371e41\") "
Jul 19 08:34:09.144887 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.144863    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "149d0e64-fc0d-4bb1-af7a-70611a371e41" (UID: "149d0e64-fc0d-4bb1-af7a-70611a371e41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:34:09.245771 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.245672    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/149d0e64-fc0d-4bb1-af7a-70611a371e41-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:34:09.907572 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.907542    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"
Jul 19 08:34:09.908028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.907541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh" event={"ID":"149d0e64-fc0d-4bb1-af7a-70611a371e41","Type":"ContainerDied","Data":"c25271a5e860026f3f151a08a9da83f87da0eb5e908631fd39739955c003f17a"}
Jul 19 08:34:09.908028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.907692    2559 scope.go:117] "RemoveContainer" containerID="4cf6acac509c31a781a97dcafa148bdbfd7c2f6d05bf4ade980cd9ac04c4e1a9"
Jul 19 08:34:09.936818 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.936782    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:34:09.939155 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:09.939131    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-hm6jh"]
Jul 19 08:34:10.460858 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:10.460823    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" path="/var/lib/kubelet/pods/149d0e64-fc0d-4bb1-af7a-70611a371e41/volumes"
Jul 19 08:34:17.715520 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715472    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715882    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715893    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715903    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715909    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715976    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.716031 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.715985    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:17.720637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.720620    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:17.725692 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.725664    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:17.816244 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.816210    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv\" (UID: \"d2b8e77f-f58e-4112-b058-27227b437178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:17.917021 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.916982    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv\" (UID: \"d2b8e77f-f58e-4112-b058-27227b437178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:17.917613 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:17.917592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv\" (UID: \"d2b8e77f-f58e-4112-b058-27227b437178\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:18.032368 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:18.032274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:18.369481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:18.369450    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:18.370592 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:34:18.370568    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b8e77f_f58e_4112_b058_27227b437178.slice/crio-d35b3e83f545ce2793bff8bc2c681d3fc0be497d333fcb596be650f7fa1661f1 WatchSource:0}: Error finding container d35b3e83f545ce2793bff8bc2c681d3fc0be497d333fcb596be650f7fa1661f1: Status 404 returned error can't find the container with id d35b3e83f545ce2793bff8bc2c681d3fc0be497d333fcb596be650f7fa1661f1
Jul 19 08:34:18.964357 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:18.964252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerStarted","Data":"958c5a754fdc432a61690a22dcfaedc72f70340546bfb0f12818b1cd5b1b7ddf"}
Jul 19 08:34:18.964357 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:18.964316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerStarted","Data":"d35b3e83f545ce2793bff8bc2c681d3fc0be497d333fcb596be650f7fa1661f1"}
Jul 19 08:34:18.979067 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:18.979018    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" podStartSLOduration=1.979003597 podStartE2EDuration="1.979003597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:18.97707 +0000 UTC m=+3095.486667900" watchObservedRunningTime="2026-07-19 08:34:18.979003597 +0000 UTC m=+3095.488601495"
Jul 19 08:34:19.971596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:19.971568    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2b8e77f-f58e-4112-b058-27227b437178" containerID="958c5a754fdc432a61690a22dcfaedc72f70340546bfb0f12818b1cd5b1b7ddf" exitCode=6
Jul 19 08:34:19.972010 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:19.971615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerDied","Data":"958c5a754fdc432a61690a22dcfaedc72f70340546bfb0f12818b1cd5b1b7ddf"}
Jul 19 08:34:19.972010 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:19.971832    2559 scope.go:117] "RemoveContainer" containerID="958c5a754fdc432a61690a22dcfaedc72f70340546bfb0f12818b1cd5b1b7ddf"
Jul 19 08:34:20.455101 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.455068    2559 scope.go:117] "RemoveContainer" containerID="38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de"
Jul 19 08:34:20.979087 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.978998    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2b8e77f-f58e-4112-b058-27227b437178" containerID="224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c" exitCode=6
Jul 19 08:34:20.979087 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.979042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerDied","Data":"224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c"}
Jul 19 08:34:20.979579 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.979140    2559 scope.go:117] "RemoveContainer" containerID="958c5a754fdc432a61690a22dcfaedc72f70340546bfb0f12818b1cd5b1b7ddf"
Jul 19 08:34:20.979579 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.979342    2559 scope.go:117] "RemoveContainer" containerID="224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c"
Jul 19 08:34:20.979579 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:20.979556    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv_odh-ai-gateway-infra(d2b8e77f-f58e-4112-b058-27227b437178)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" podUID="d2b8e77f-f58e-4112-b058-27227b437178"
Jul 19 08:34:20.981071 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:20.981051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerStarted","Data":"818dfa5175c830cd00b7d394462e667df5c8c4dd6f6c753e09e8efdf2a4a137c"}
Jul 19 08:34:21.010141 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.010099    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" podStartSLOduration=16.010080885 podStartE2EDuration="16.010080885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:21.007635449 +0000 UTC m=+3097.517233348" watchObservedRunningTime="2026-07-19 08:34:21.010080885 +0000 UTC m=+3097.519678786"
Jul 19 08:34:21.988504 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.988470    2559 scope.go:117] "RemoveContainer" containerID="224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c"
Jul 19 08:34:21.988947 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:21.988738    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv_odh-ai-gateway-infra(d2b8e77f-f58e-4112-b058-27227b437178)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" podUID="d2b8e77f-f58e-4112-b058-27227b437178"
Jul 19 08:34:21.989986 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.989964    2559 generic.go:356] "Generic (PLEG): container finished" podID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerID="818dfa5175c830cd00b7d394462e667df5c8c4dd6f6c753e09e8efdf2a4a137c" exitCode=6
Jul 19 08:34:21.990072 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.990059    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerDied","Data":"818dfa5175c830cd00b7d394462e667df5c8c4dd6f6c753e09e8efdf2a4a137c"}
Jul 19 08:34:21.990123 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.990111    2559 scope.go:117] "RemoveContainer" containerID="38fc80c5480e9e341444bfe06d2db3779de5eb7728d2a8dc9ddb3e6c4eaeb3de"
Jul 19 08:34:21.990342 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:21.990327    2559 scope.go:117] "RemoveContainer" containerID="818dfa5175c830cd00b7d394462e667df5c8c4dd6f6c753e09e8efdf2a4a137c"
Jul 19 08:34:21.990519 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:21.990501    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l_odh-ai-gateway-infra(38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8"
Jul 19 08:34:22.017802 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:22.017772    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:23.137166 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:23.137139    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:23.163341 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:23.163316    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca\") pod \"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8\" (UID: \"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8\") "
Jul 19 08:34:23.163747 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:23.163722    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" (UID: "38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:34:23.264446 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:23.264358    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:34:24.002683 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.002655    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"
Jul 19 08:34:24.002915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.002721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l" event={"ID":"38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8","Type":"ContainerDied","Data":"0e4441e464011272872a5603ba33a5ca4d526b7487514dfc80b12df14f4cfe95"}
Jul 19 08:34:24.002915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.002793    2559 scope.go:117] "RemoveContainer" containerID="818dfa5175c830cd00b7d394462e667df5c8c4dd6f6c753e09e8efdf2a4a137c"
Jul 19 08:34:24.023906 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.023881    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:24.026182 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.026163    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-9pf8l"]
Jul 19 08:34:24.461022 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:24.460992    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" path="/var/lib/kubelet/pods/38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8/volumes"
Jul 19 08:34:34.457695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:34.457660    2559 scope.go:117] "RemoveContainer" containerID="224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c"
Jul 19 08:34:35.068296 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:35.068264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerStarted","Data":"09fb731634ef7769457be2643b949824d52e49f0603e6cb6e12d40f0b171bbd5"}
Jul 19 08:34:36.075112 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:36.075075    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2b8e77f-f58e-4112-b058-27227b437178" containerID="09fb731634ef7769457be2643b949824d52e49f0603e6cb6e12d40f0b171bbd5" exitCode=6
Jul 19 08:34:36.075533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:36.075143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerDied","Data":"09fb731634ef7769457be2643b949824d52e49f0603e6cb6e12d40f0b171bbd5"}
Jul 19 08:34:36.075533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:36.075245    2559 scope.go:117] "RemoveContainer" containerID="224f53401d8f8e6eebb9621e4508544af76b2e01f49678d0f554a12439e48a5c"
Jul 19 08:34:36.075649 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:36.075580    2559 scope.go:117] "RemoveContainer" containerID="09fb731634ef7769457be2643b949824d52e49f0603e6cb6e12d40f0b171bbd5"
Jul 19 08:34:36.075855 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:36.075830    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv_odh-ai-gateway-infra(d2b8e77f-f58e-4112-b058-27227b437178)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" podUID="d2b8e77f-f58e-4112-b058-27227b437178"
Jul 19 08:34:36.091288 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:36.091263    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:37.214621 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:37.214595    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:37.289823 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:37.289793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca\") pod \"d2b8e77f-f58e-4112-b058-27227b437178\" (UID: \"d2b8e77f-f58e-4112-b058-27227b437178\") "
Jul 19 08:34:37.290148 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:37.290124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2b8e77f-f58e-4112-b058-27227b437178" (UID: "d2b8e77f-f58e-4112-b058-27227b437178"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:34:37.390885 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:37.390815    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b8e77f-f58e-4112-b058-27227b437178-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:34:38.088526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.088500    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"
Jul 19 08:34:38.088686 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.088530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv" event={"ID":"d2b8e77f-f58e-4112-b058-27227b437178","Type":"ContainerDied","Data":"d35b3e83f545ce2793bff8bc2c681d3fc0be497d333fcb596be650f7fa1661f1"}
Jul 19 08:34:38.088686 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.088594    2559 scope.go:117] "RemoveContainer" containerID="09fb731634ef7769457be2643b949824d52e49f0603e6cb6e12d40f0b171bbd5"
Jul 19 08:34:38.110156 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.110129    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:38.114105 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.114083    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-nz5dv"]
Jul 19 08:34:38.460712 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:38.460616    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b8e77f-f58e-4112-b058-27227b437178" path="/var/lib/kubelet/pods/d2b8e77f-f58e-4112-b058-27227b437178/volumes"
Jul 19 08:34:40.480044 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480012    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:34:40.480396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480374    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480390    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480407    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480413    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480420    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480426    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480438    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480444    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480450    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480455    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480462    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480467    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480517    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480524    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480530    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="149d0e64-fc0d-4bb1-af7a-70611a371e41" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480536    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480543    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.480815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.480550    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:40.483634 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.483617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:34:40.486440 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.486414    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:34:40.486585 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.486455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:34:40.489727 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.489690    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:34:40.623268 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.623235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r\" (UID: \"dd729251-233f-43dc-b155-6e73ff3c08a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:34:40.724475 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.724442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r\" (UID: \"dd729251-233f-43dc-b155-6e73ff3c08a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:34:40.725088 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.725070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r\" (UID: \"dd729251-233f-43dc-b155-6e73ff3c08a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:34:40.796357 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.796279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:34:40.927231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:40.927205    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:34:40.929143 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:34:40.929120    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd729251_233f_43dc_b155_6e73ff3c08a3.slice/crio-73a28c9c5f70f1c7722277ae21158960074b8f5fa90c7c09c6803d3d3edacc26 WatchSource:0}: Error finding container 73a28c9c5f70f1c7722277ae21158960074b8f5fa90c7c09c6803d3d3edacc26: Status 404 returned error can't find the container with id 73a28c9c5f70f1c7722277ae21158960074b8f5fa90c7c09c6803d3d3edacc26
Jul 19 08:34:41.106006 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:41.105975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" event={"ID":"dd729251-233f-43dc-b155-6e73ff3c08a3","Type":"ContainerStarted","Data":"73a28c9c5f70f1c7722277ae21158960074b8f5fa90c7c09c6803d3d3edacc26"}
Jul 19 08:34:42.114029 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:42.113990    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerID="c08e35533c1049e7b070d8f3083ca03397c4b9dce5d8770e413c4839e3686df6" exitCode=6
Jul 19 08:34:42.114414 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:42.114066    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" event={"ID":"dd729251-233f-43dc-b155-6e73ff3c08a3","Type":"ContainerDied","Data":"c08e35533c1049e7b070d8f3083ca03397c4b9dce5d8770e413c4839e3686df6"}
Jul 19 08:34:42.114414 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:42.114247    2559 scope.go:117] "RemoveContainer" containerID="c08e35533c1049e7b070d8f3083ca03397c4b9dce5d8770e413c4839e3686df6"
Jul 19 08:34:43.121264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.121231    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerID="daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038" exitCode=6
Jul 19 08:34:43.121654 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.121323    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" event={"ID":"dd729251-233f-43dc-b155-6e73ff3c08a3","Type":"ContainerDied","Data":"daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038"}
Jul 19 08:34:43.121654 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.121385    2559 scope.go:117] "RemoveContainer" containerID="c08e35533c1049e7b070d8f3083ca03397c4b9dce5d8770e413c4839e3686df6"
Jul 19 08:34:43.121654 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.121526    2559 scope.go:117] "RemoveContainer" containerID="daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038"
Jul 19 08:34:43.121820 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:43.121799    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r_odh-ai-gateway-infra(dd729251-233f-43dc-b155-6e73ff3c08a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3"
Jul 19 08:34:43.722116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.722082    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:34:43.722485 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.722472    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:43.722533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.722487    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b8e77f-f58e-4112-b058-27227b437178" containerName="revoke-keys"
Jul 19 08:34:43.722568 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.722554    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f6bc16-f0a3-40d9-8f64-9cbd7b8362f8" containerName="revoke-keys"
Jul 19 08:34:43.726768 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.726752    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:43.732196 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.732165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:34:43.853403 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.853359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d\" (UID: \"6220131f-f570-4c17-81db-d5ded337f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:43.953860 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.953816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d\" (UID: \"6220131f-f570-4c17-81db-d5ded337f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:43.954476 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:43.954453    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d\" (UID: \"6220131f-f570-4c17-81db-d5ded337f62f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:44.039121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:44.039038    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:44.130732 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:44.130535    2559 scope.go:117] "RemoveContainer" containerID="daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038"
Jul 19 08:34:44.131157 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:44.130809    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r_odh-ai-gateway-infra(dd729251-233f-43dc-b155-6e73ff3c08a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3"
Jul 19 08:34:44.385257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:44.385235    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:34:44.386402 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:34:44.386355    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6220131f_f570_4c17_81db_d5ded337f62f.slice/crio-43a6c6d3b9e4ca65bf97b95870b680fe2622fc537644ba84d581f31f8c575949 WatchSource:0}: Error finding container 43a6c6d3b9e4ca65bf97b95870b680fe2622fc537644ba84d581f31f8c575949: Status 404 returned error can't find the container with id 43a6c6d3b9e4ca65bf97b95870b680fe2622fc537644ba84d581f31f8c575949
Jul 19 08:34:45.136836 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:45.136801    2559 generic.go:356] "Generic (PLEG): container finished" podID="6220131f-f570-4c17-81db-d5ded337f62f" containerID="65a1d4912bcc4f1610f9468f0e6943ca5493b6e5ac00aad0866eaeeb366c230b" exitCode=6
Jul 19 08:34:45.137220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:45.136871    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" event={"ID":"6220131f-f570-4c17-81db-d5ded337f62f","Type":"ContainerDied","Data":"65a1d4912bcc4f1610f9468f0e6943ca5493b6e5ac00aad0866eaeeb366c230b"}
Jul 19 08:34:45.137220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:45.136915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" event={"ID":"6220131f-f570-4c17-81db-d5ded337f62f","Type":"ContainerStarted","Data":"43a6c6d3b9e4ca65bf97b95870b680fe2622fc537644ba84d581f31f8c575949"}
Jul 19 08:34:45.137220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:45.137102    2559 scope.go:117] "RemoveContainer" containerID="65a1d4912bcc4f1610f9468f0e6943ca5493b6e5ac00aad0866eaeeb366c230b"
Jul 19 08:34:46.147520 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:46.147489    2559 generic.go:356] "Generic (PLEG): container finished" podID="6220131f-f570-4c17-81db-d5ded337f62f" containerID="a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943" exitCode=6
Jul 19 08:34:46.147951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:46.147578    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" event={"ID":"6220131f-f570-4c17-81db-d5ded337f62f","Type":"ContainerDied","Data":"a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943"}
Jul 19 08:34:46.147951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:46.147648    2559 scope.go:117] "RemoveContainer" containerID="65a1d4912bcc4f1610f9468f0e6943ca5493b6e5ac00aad0866eaeeb366c230b"
Jul 19 08:34:46.147951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:46.147818    2559 scope.go:117] "RemoveContainer" containerID="a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943"
Jul 19 08:34:46.148087 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:46.148015    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d_odh-ai-gateway-infra(6220131f-f570-4c17-81db-d5ded337f62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" podUID="6220131f-f570-4c17-81db-d5ded337f62f"
Jul 19 08:34:47.154466 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:47.154435    2559 scope.go:117] "RemoveContainer" containerID="a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943"
Jul 19 08:34:47.154919 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:47.154690    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d_odh-ai-gateway-infra(6220131f-f570-4c17-81db-d5ded337f62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" podUID="6220131f-f570-4c17-81db-d5ded337f62f"
Jul 19 08:34:55.419117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.419078    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:34:55.422604 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.422584    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:34:55.429587 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.429555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:34:55.566019 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.565982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ck477\" (UID: \"da3b4698-717d-4f3b-9958-42708a6614dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:34:55.667285 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.667245    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ck477\" (UID: \"da3b4698-717d-4f3b-9958-42708a6614dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:34:55.667943 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.667925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-ck477\" (UID: \"da3b4698-717d-4f3b-9958-42708a6614dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:34:55.734926 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.734859    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:34:55.864042 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:55.864018    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:34:55.865836 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:34:55.865804    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3b4698_717d_4f3b_9958_42708a6614dc.slice/crio-b999ad8d7353d02e7250fc0b10d25da7fe7efa5521e6127d7868842a5c8d8ac2 WatchSource:0}: Error finding container b999ad8d7353d02e7250fc0b10d25da7fe7efa5521e6127d7868842a5c8d8ac2: Status 404 returned error can't find the container with id b999ad8d7353d02e7250fc0b10d25da7fe7efa5521e6127d7868842a5c8d8ac2
Jul 19 08:34:56.205776 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:56.205741    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerStarted","Data":"033f9249ce2a398ae7287cb5ef92da4e0baaf8b28baf671091af1eadfa8a3be6"}
Jul 19 08:34:56.205910 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:56.205797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerStarted","Data":"b999ad8d7353d02e7250fc0b10d25da7fe7efa5521e6127d7868842a5c8d8ac2"}
Jul 19 08:34:56.220402 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:56.220359    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" podStartSLOduration=1.220344915 podStartE2EDuration="1.220344915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:34:56.219347607 +0000 UTC m=+3132.728945508" watchObservedRunningTime="2026-07-19 08:34:56.220344915 +0000 UTC m=+3132.729942814"
Jul 19 08:34:57.212087 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:57.212051    2559 generic.go:356] "Generic (PLEG): container finished" podID="da3b4698-717d-4f3b-9958-42708a6614dc" containerID="033f9249ce2a398ae7287cb5ef92da4e0baaf8b28baf671091af1eadfa8a3be6" exitCode=6
Jul 19 08:34:57.212478 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:57.212134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerDied","Data":"033f9249ce2a398ae7287cb5ef92da4e0baaf8b28baf671091af1eadfa8a3be6"}
Jul 19 08:34:57.212478 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:57.212284    2559 scope.go:117] "RemoveContainer" containerID="033f9249ce2a398ae7287cb5ef92da4e0baaf8b28baf671091af1eadfa8a3be6"
Jul 19 08:34:57.454997 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:57.454970    2559 scope.go:117] "RemoveContainer" containerID="a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943"
Jul 19 08:34:58.220209 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.220177    2559 generic.go:356] "Generic (PLEG): container finished" podID="6220131f-f570-4c17-81db-d5ded337f62f" containerID="6b17763fbf565c45f475f6eb839d30994a7b87552ca2eb92afc54556bd6c6ea4" exitCode=6
Jul 19 08:34:58.220668 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.220254    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" event={"ID":"6220131f-f570-4c17-81db-d5ded337f62f","Type":"ContainerDied","Data":"6b17763fbf565c45f475f6eb839d30994a7b87552ca2eb92afc54556bd6c6ea4"}
Jul 19 08:34:58.220668 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.220317    2559 scope.go:117] "RemoveContainer" containerID="a5adf9ba0ba08561bf97a91ed272fd4f160aa3999bae8f6b81ba6022d99ea943"
Jul 19 08:34:58.220823 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.220682    2559 scope.go:117] "RemoveContainer" containerID="6b17763fbf565c45f475f6eb839d30994a7b87552ca2eb92afc54556bd6c6ea4"
Jul 19 08:34:58.220961 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:58.220940    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d_odh-ai-gateway-infra(6220131f-f570-4c17-81db-d5ded337f62f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" podUID="6220131f-f570-4c17-81db-d5ded337f62f"
Jul 19 08:34:58.222290 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.222271    2559 generic.go:356] "Generic (PLEG): container finished" podID="da3b4698-717d-4f3b-9958-42708a6614dc" containerID="849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd" exitCode=6
Jul 19 08:34:58.222379 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.222324    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerDied","Data":"849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd"}
Jul 19 08:34:58.222485 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.222471    2559 scope.go:117] "RemoveContainer" containerID="849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd"
Jul 19 08:34:58.222620 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:58.222604    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ck477_odh-ai-gateway-infra(da3b4698-717d-4f3b-9958-42708a6614dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" podUID="da3b4698-717d-4f3b-9958-42708a6614dc"
Jul 19 08:34:58.231572 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.231557    2559 scope.go:117] "RemoveContainer" containerID="033f9249ce2a398ae7287cb5ef92da4e0baaf8b28baf671091af1eadfa8a3be6"
Jul 19 08:34:58.455204 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:58.455169    2559 scope.go:117] "RemoveContainer" containerID="daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038"
Jul 19 08:34:59.230570 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.230544    2559 scope.go:117] "RemoveContainer" containerID="849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd"
Jul 19 08:34:59.230988 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:59.230824    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ck477_odh-ai-gateway-infra(da3b4698-717d-4f3b-9958-42708a6614dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" podUID="da3b4698-717d-4f3b-9958-42708a6614dc"
Jul 19 08:34:59.232358 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.232335    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerID="3d2597d1072088eb2b57269925ba7e00bafa56e81fd01b399f1ee91fd71895e9" exitCode=6
Jul 19 08:34:59.232479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.232408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" event={"ID":"dd729251-233f-43dc-b155-6e73ff3c08a3","Type":"ContainerDied","Data":"3d2597d1072088eb2b57269925ba7e00bafa56e81fd01b399f1ee91fd71895e9"}
Jul 19 08:34:59.232546 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.232502    2559 scope.go:117] "RemoveContainer" containerID="daf894968ccd657b632475e1cce11b93d511fdfe38c35183d1abc18db1616038"
Jul 19 08:34:59.232821 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.232805    2559 scope.go:117] "RemoveContainer" containerID="3d2597d1072088eb2b57269925ba7e00bafa56e81fd01b399f1ee91fd71895e9"
Jul 19 08:34:59.233006 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:34:59.232986    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r_odh-ai-gateway-infra(dd729251-233f-43dc-b155-6e73ff3c08a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3"
Jul 19 08:34:59.246326 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.246303    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:34:59.384815 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.384790    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:34:59.408600 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.408529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca\") pod \"6220131f-f570-4c17-81db-d5ded337f62f\" (UID: \"6220131f-f570-4c17-81db-d5ded337f62f\") "
Jul 19 08:34:59.408878 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.408856    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6220131f-f570-4c17-81db-d5ded337f62f" (UID: "6220131f-f570-4c17-81db-d5ded337f62f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:34:59.509574 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:34:59.509538    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6220131f-f570-4c17-81db-d5ded337f62f-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:00.239243 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.239213    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"
Jul 19 08:35:00.239733 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.239251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d" event={"ID":"6220131f-f570-4c17-81db-d5ded337f62f","Type":"ContainerDied","Data":"43a6c6d3b9e4ca65bf97b95870b680fe2622fc537644ba84d581f31f8c575949"}
Jul 19 08:35:00.239733 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.239328    2559 scope.go:117] "RemoveContainer" containerID="6b17763fbf565c45f475f6eb839d30994a7b87552ca2eb92afc54556bd6c6ea4"
Jul 19 08:35:00.263011 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.262984    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:35:00.269238 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.269213    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-9dd8d"]
Jul 19 08:35:00.271647 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.271623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:35:00.402260 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.402236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:35:00.415522 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.415496    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca\") pod \"dd729251-233f-43dc-b155-6e73ff3c08a3\" (UID: \"dd729251-233f-43dc-b155-6e73ff3c08a3\") "
Jul 19 08:35:00.415891 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.415868    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd729251-233f-43dc-b155-6e73ff3c08a3" (UID: "dd729251-233f-43dc-b155-6e73ff3c08a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:35:00.461311 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.461277    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6220131f-f570-4c17-81db-d5ded337f62f" path="/var/lib/kubelet/pods/6220131f-f570-4c17-81db-d5ded337f62f/volumes"
Jul 19 08:35:00.516687 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:00.516605    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd729251-233f-43dc-b155-6e73ff3c08a3-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:01.248241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:01.248205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r" event={"ID":"dd729251-233f-43dc-b155-6e73ff3c08a3","Type":"ContainerDied","Data":"73a28c9c5f70f1c7722277ae21158960074b8f5fa90c7c09c6803d3d3edacc26"}
Jul 19 08:35:01.248656 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:01.248264    2559 scope.go:117] "RemoveContainer" containerID="3d2597d1072088eb2b57269925ba7e00bafa56e81fd01b399f1ee91fd71895e9"
Jul 19 08:35:01.248656 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:01.248214    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"
Jul 19 08:35:01.264811 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:01.264788    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:35:01.268001 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:01.267977    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-8849r"]
Jul 19 08:35:02.460231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:02.460197    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" path="/var/lib/kubelet/pods/dd729251-233f-43dc-b155-6e73ff3c08a3/volumes"
Jul 19 08:35:14.458377 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:14.458284    2559 scope.go:117] "RemoveContainer" containerID="849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd"
Jul 19 08:35:15.326107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:15.326025    2559 generic.go:356] "Generic (PLEG): container finished" podID="da3b4698-717d-4f3b-9958-42708a6614dc" containerID="6ec6849a40306a08482eb19ce19965414dd481d63320e432f39b1908ee932b0a" exitCode=6
Jul 19 08:35:15.326107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:15.326062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerDied","Data":"6ec6849a40306a08482eb19ce19965414dd481d63320e432f39b1908ee932b0a"}
Jul 19 08:35:15.326287 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:15.326118    2559 scope.go:117] "RemoveContainer" containerID="849fdc936786e9e30d8a77393c5a158dc9f8481c2af09dad04d6cc5e3ec22bbd"
Jul 19 08:35:15.326432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:15.326415    2559 scope.go:117] "RemoveContainer" containerID="6ec6849a40306a08482eb19ce19965414dd481d63320e432f39b1908ee932b0a"
Jul 19 08:35:15.326641 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:15.326618    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-ck477_odh-ai-gateway-infra(da3b4698-717d-4f3b-9958-42708a6614dc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" podUID="da3b4698-717d-4f3b-9958-42708a6614dc"
Jul 19 08:35:16.351524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:16.351489    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:35:16.491243 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:16.491217    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:35:16.556903 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:16.556870    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca\") pod \"da3b4698-717d-4f3b-9958-42708a6614dc\" (UID: \"da3b4698-717d-4f3b-9958-42708a6614dc\") "
Jul 19 08:35:16.557241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:16.557218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da3b4698-717d-4f3b-9958-42708a6614dc" (UID: "da3b4698-717d-4f3b-9958-42708a6614dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:35:16.657570 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:16.657495    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da3b4698-717d-4f3b-9958-42708a6614dc-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:17.340545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.340510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477" event={"ID":"da3b4698-717d-4f3b-9958-42708a6614dc","Type":"ContainerDied","Data":"b999ad8d7353d02e7250fc0b10d25da7fe7efa5521e6127d7868842a5c8d8ac2"}
Jul 19 08:35:17.340741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.340564    2559 scope.go:117] "RemoveContainer" containerID="6ec6849a40306a08482eb19ce19965414dd481d63320e432f39b1908ee932b0a"
Jul 19 08:35:17.340741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.340517    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"
Jul 19 08:35:17.362268 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.362243    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:35:17.367396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.367376    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-ck477"]
Jul 19 08:35:17.759138 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759051    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:17.759421 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759408    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759422    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759432    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759438    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759447    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759453    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759464    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759469    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759483    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759489    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759496    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759501    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759556    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759562    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759569    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759576    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759581    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759586    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:17.759678 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.759593    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:17.763862 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.763846    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:17.766481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.766439    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:35:17.766481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.766470    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:35:17.769741 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.769717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:17.869868 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.869832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-797hv\" (UID: \"6313da22-6ae2-484f-a40d-603f7d6a2b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:17.971210 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.971163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-797hv\" (UID: \"6313da22-6ae2-484f-a40d-603f7d6a2b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:17.971821 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:17.971800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-797hv\" (UID: \"6313da22-6ae2-484f-a40d-603f7d6a2b2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:18.075296 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:18.075268    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:18.206193 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:18.206165    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:18.207918 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:35:18.207892    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6313da22_6ae2_484f_a40d_603f7d6a2b2f.slice/crio-e6a5972293287eb0bdf5b63eb40acc7682877cd1d830a8af5002f1f07aeefd53 WatchSource:0}: Error finding container e6a5972293287eb0bdf5b63eb40acc7682877cd1d830a8af5002f1f07aeefd53: Status 404 returned error can't find the container with id e6a5972293287eb0bdf5b63eb40acc7682877cd1d830a8af5002f1f07aeefd53
Jul 19 08:35:18.348246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:18.348162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" event={"ID":"6313da22-6ae2-484f-a40d-603f7d6a2b2f","Type":"ContainerStarted","Data":"e6a5972293287eb0bdf5b63eb40acc7682877cd1d830a8af5002f1f07aeefd53"}
Jul 19 08:35:18.462461 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:18.462429    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" path="/var/lib/kubelet/pods/da3b4698-717d-4f3b-9958-42708a6614dc/volumes"
Jul 19 08:35:19.357057 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:19.357023    2559 generic.go:356] "Generic (PLEG): container finished" podID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerID="9f963576ada41ee5bcac1af7a02161a8a8afc808f94b8deeaa64415208ecd089" exitCode=6
Jul 19 08:35:19.357234 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:19.357113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" event={"ID":"6313da22-6ae2-484f-a40d-603f7d6a2b2f","Type":"ContainerDied","Data":"9f963576ada41ee5bcac1af7a02161a8a8afc808f94b8deeaa64415208ecd089"}
Jul 19 08:35:19.357290 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:19.357274    2559 scope.go:117] "RemoveContainer" containerID="9f963576ada41ee5bcac1af7a02161a8a8afc808f94b8deeaa64415208ecd089"
Jul 19 08:35:20.364864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:20.364832    2559 generic.go:356] "Generic (PLEG): container finished" podID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerID="7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099" exitCode=6
Jul 19 08:35:20.365331 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:20.364921    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" event={"ID":"6313da22-6ae2-484f-a40d-603f7d6a2b2f","Type":"ContainerDied","Data":"7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099"}
Jul 19 08:35:20.365331 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:20.365000    2559 scope.go:117] "RemoveContainer" containerID="9f963576ada41ee5bcac1af7a02161a8a8afc808f94b8deeaa64415208ecd089"
Jul 19 08:35:20.365331 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:20.365119    2559 scope.go:117] "RemoveContainer" containerID="7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099"
Jul 19 08:35:20.365331 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:20.365318    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-797hv_odh-ai-gateway-infra(6313da22-6ae2-484f-a40d-603f7d6a2b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f"
Jul 19 08:35:21.372097 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:21.372069    2559 scope.go:117] "RemoveContainer" containerID="7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099"
Jul 19 08:35:21.372443 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:21.372284    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-797hv_odh-ai-gateway-infra(6313da22-6ae2-484f-a40d-603f7d6a2b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f"
Jul 19 08:35:33.768268 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768235    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768596    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768607    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768616    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da3b4698-717d-4f3b-9958-42708a6614dc" containerName="revoke-keys"
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768630    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:33.768688 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768636    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:33.768913 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768694    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd729251-233f-43dc-b155-6e73ff3c08a3" containerName="revoke-keys"
Jul 19 08:35:33.768913 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.768716    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6220131f-f570-4c17-81db-d5ded337f62f" containerName="revoke-keys"
Jul 19 08:35:33.771992 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.771976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:33.780250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.780221    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:33.922179 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:33.922146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm\" (UID: \"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:34.022827 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:34.022739    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm\" (UID: \"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:34.023385 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:34.023366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm\" (UID: \"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:34.083756 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:34.083694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:34.218762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:34.218737    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:34.221302 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:35:34.221273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78b2a6a_cb3a_4fa2_a0e4_f2f462311ed4.slice/crio-893bda482c03f8a721efd069ec87a8bbd25304168b49cf27cd922264aac070f8 WatchSource:0}: Error finding container 893bda482c03f8a721efd069ec87a8bbd25304168b49cf27cd922264aac070f8: Status 404 returned error can't find the container with id 893bda482c03f8a721efd069ec87a8bbd25304168b49cf27cd922264aac070f8
Jul 19 08:35:34.448126 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:34.448085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" event={"ID":"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4","Type":"ContainerStarted","Data":"893bda482c03f8a721efd069ec87a8bbd25304168b49cf27cd922264aac070f8"}
Jul 19 08:35:35.454757 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:35.454725    2559 scope.go:117] "RemoveContainer" containerID="7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099"
Jul 19 08:35:35.455778 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:35.455748    2559 generic.go:356] "Generic (PLEG): container finished" podID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerID="82b1a6be3f1057b3ef426c09a6e647e2a46eb160d00f296f4b4d10f7984e2230" exitCode=6
Jul 19 08:35:35.455911 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:35.455786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" event={"ID":"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4","Type":"ContainerDied","Data":"82b1a6be3f1057b3ef426c09a6e647e2a46eb160d00f296f4b4d10f7984e2230"}
Jul 19 08:35:35.456012 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:35.455991    2559 scope.go:117] "RemoveContainer" containerID="82b1a6be3f1057b3ef426c09a6e647e2a46eb160d00f296f4b4d10f7984e2230"
Jul 19 08:35:36.463593 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.463508    2559 generic.go:356] "Generic (PLEG): container finished" podID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerID="64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625" exitCode=6
Jul 19 08:35:36.464059 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.463593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" event={"ID":"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4","Type":"ContainerDied","Data":"64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625"}
Jul 19 08:35:36.464059 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.463649    2559 scope.go:117] "RemoveContainer" containerID="82b1a6be3f1057b3ef426c09a6e647e2a46eb160d00f296f4b4d10f7984e2230"
Jul 19 08:35:36.464059 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.463885    2559 scope.go:117] "RemoveContainer" containerID="64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625"
Jul 19 08:35:36.464230 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:36.464101    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm_odh-ai-gateway-infra(d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4"
Jul 19 08:35:36.465479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.465461    2559 generic.go:356] "Generic (PLEG): container finished" podID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerID="56f7e8a7f2cc1d5cde5bd7e89dacd5a093f961eca3b7172e9f3df4f742a8d6d2" exitCode=6
Jul 19 08:35:36.465549 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.465529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" event={"ID":"6313da22-6ae2-484f-a40d-603f7d6a2b2f","Type":"ContainerDied","Data":"56f7e8a7f2cc1d5cde5bd7e89dacd5a093f961eca3b7172e9f3df4f742a8d6d2"}
Jul 19 08:35:36.465833 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.465820    2559 scope.go:117] "RemoveContainer" containerID="56f7e8a7f2cc1d5cde5bd7e89dacd5a093f961eca3b7172e9f3df4f742a8d6d2"
Jul 19 08:35:36.465998 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:36.465981    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-797hv_odh-ai-gateway-infra(6313da22-6ae2-484f-a40d-603f7d6a2b2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f"
Jul 19 08:35:36.474871 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:36.474855    2559 scope.go:117] "RemoveContainer" containerID="7f247327f0194a73bddb3b7306dcd3beec3fdaec619af08f54d8c3450872a099"
Jul 19 08:35:37.472503 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.472478    2559 scope.go:117] "RemoveContainer" containerID="64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625"
Jul 19 08:35:37.472938 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:37.472731    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm_odh-ai-gateway-infra(d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4"
Jul 19 08:35:37.499451 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.499426    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:37.636711 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.636672    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:37.753203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.753121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca\") pod \"6313da22-6ae2-484f-a40d-603f7d6a2b2f\" (UID: \"6313da22-6ae2-484f-a40d-603f7d6a2b2f\") "
Jul 19 08:35:37.753547 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.753521    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6313da22-6ae2-484f-a40d-603f7d6a2b2f" (UID: "6313da22-6ae2-484f-a40d-603f7d6a2b2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:35:37.854298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:37.854265    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6313da22-6ae2-484f-a40d-603f7d6a2b2f-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:38.479770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:38.479746    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"
Jul 19 08:35:38.479770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:38.479756    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv" event={"ID":"6313da22-6ae2-484f-a40d-603f7d6a2b2f","Type":"ContainerDied","Data":"e6a5972293287eb0bdf5b63eb40acc7682877cd1d830a8af5002f1f07aeefd53"}
Jul 19 08:35:38.480259 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:38.479813    2559 scope.go:117] "RemoveContainer" containerID="56f7e8a7f2cc1d5cde5bd7e89dacd5a093f961eca3b7172e9f3df4f742a8d6d2"
Jul 19 08:35:38.496450 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:38.496405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:38.498957 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:38.498937    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-797hv"]
Jul 19 08:35:40.461593 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.461564    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" path="/var/lib/kubelet/pods/6313da22-6ae2-484f-a40d-603f7d6a2b2f/volumes"
Jul 19 08:35:40.521717 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.521678    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:35:40.522119 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522102    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522122    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522141    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522148    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522162    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522203 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522172    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522453 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522287    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522453 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522302    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.522453 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.522312    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6313da22-6ae2-484f-a40d-603f7d6a2b2f" containerName="revoke-keys"
Jul 19 08:35:40.526913 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.526893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:40.531787 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.531756    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:35:40.680404 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.680373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z\" (UID: \"8104b6c9-880d-4225-82bc-6d132208d7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:40.781925 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.781849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z\" (UID: \"8104b6c9-880d-4225-82bc-6d132208d7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:40.782491 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.782470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z\" (UID: \"8104b6c9-880d-4225-82bc-6d132208d7ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:40.838933 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.838903    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:40.967610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:40.967581    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:35:40.968681 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:35:40.968649    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8104b6c9_880d_4225_82bc_6d132208d7ee.slice/crio-2e30403b89021f3b46dfd863e2c64a9783a5b982bbe95025f45e4a8bf852809b WatchSource:0}: Error finding container 2e30403b89021f3b46dfd863e2c64a9783a5b982bbe95025f45e4a8bf852809b: Status 404 returned error can't find the container with id 2e30403b89021f3b46dfd863e2c64a9783a5b982bbe95025f45e4a8bf852809b
Jul 19 08:35:41.499625 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:41.499526    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerStarted","Data":"2c50fd4500b051fd8ac05d6c0f9c2cb611c768b51b21444836cc8b170baf753f"}
Jul 19 08:35:41.499625 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:41.499571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerStarted","Data":"2e30403b89021f3b46dfd863e2c64a9783a5b982bbe95025f45e4a8bf852809b"}
Jul 19 08:35:41.516209 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:41.516161    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" podStartSLOduration=1.516146081 podStartE2EDuration="1.516146081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:35:41.513186307 +0000 UTC m=+3178.022784206" watchObservedRunningTime="2026-07-19 08:35:41.516146081 +0000 UTC m=+3178.025743998"
Jul 19 08:35:42.506742 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:42.506688    2559 generic.go:356] "Generic (PLEG): container finished" podID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerID="2c50fd4500b051fd8ac05d6c0f9c2cb611c768b51b21444836cc8b170baf753f" exitCode=6
Jul 19 08:35:42.506742 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:42.506734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerDied","Data":"2c50fd4500b051fd8ac05d6c0f9c2cb611c768b51b21444836cc8b170baf753f"}
Jul 19 08:35:42.507295 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:42.506964    2559 scope.go:117] "RemoveContainer" containerID="2c50fd4500b051fd8ac05d6c0f9c2cb611c768b51b21444836cc8b170baf753f"
Jul 19 08:35:43.514596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:43.514510    2559 generic.go:356] "Generic (PLEG): container finished" podID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerID="b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a" exitCode=6
Jul 19 08:35:43.514596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:43.514569    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerDied","Data":"b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a"}
Jul 19 08:35:43.515061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:43.514634    2559 scope.go:117] "RemoveContainer" containerID="2c50fd4500b051fd8ac05d6c0f9c2cb611c768b51b21444836cc8b170baf753f"
Jul 19 08:35:43.515061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:43.514863    2559 scope.go:117] "RemoveContainer" containerID="b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a"
Jul 19 08:35:43.515149 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:43.515127    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z_odh-ai-gateway-infra(8104b6c9-880d-4225-82bc-6d132208d7ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee"
Jul 19 08:35:44.522433 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:44.522406    2559 scope.go:117] "RemoveContainer" containerID="b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a"
Jul 19 08:35:44.522843 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:44.522598    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z_odh-ai-gateway-infra(8104b6c9-880d-4225-82bc-6d132208d7ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee"
Jul 19 08:35:51.455278 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:51.455249    2559 scope.go:117] "RemoveContainer" containerID="64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625"
Jul 19 08:35:52.568006 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:52.567973    2559 generic.go:356] "Generic (PLEG): container finished" podID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerID="64942e4991fb5cbe59b6fae706e79897b45f76a4d31ae179ff7dcd7eba525090" exitCode=6
Jul 19 08:35:52.568463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:52.568040    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" event={"ID":"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4","Type":"ContainerDied","Data":"64942e4991fb5cbe59b6fae706e79897b45f76a4d31ae179ff7dcd7eba525090"}
Jul 19 08:35:52.568463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:52.568097    2559 scope.go:117] "RemoveContainer" containerID="64eb20dc3ad9afc347271c6f860e061eea198e7defcf5ec97294f10ee70aa625"
Jul 19 08:35:52.568463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:52.568440    2559 scope.go:117] "RemoveContainer" containerID="64942e4991fb5cbe59b6fae706e79897b45f76a4d31ae179ff7dcd7eba525090"
Jul 19 08:35:52.568723 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:52.568676    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm_odh-ai-gateway-infra(d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4"
Jul 19 08:35:53.592214 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:53.592175    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:53.734881 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:53.734855    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:53.797689 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:53.797659    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca\") pod \"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4\" (UID: \"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4\") "
Jul 19 08:35:53.798015 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:53.797992    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" (UID: "d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:35:53.898748 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:53.898655    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:54.582661 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:54.582631    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm" event={"ID":"d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4","Type":"ContainerDied","Data":"893bda482c03f8a721efd069ec87a8bbd25304168b49cf27cd922264aac070f8"}
Jul 19 08:35:54.582661 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:54.582647    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"
Jul 19 08:35:54.582931 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:54.582722    2559 scope.go:117] "RemoveContainer" containerID="64942e4991fb5cbe59b6fae706e79897b45f76a4d31ae179ff7dcd7eba525090"
Jul 19 08:35:54.600559 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:54.600533    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:54.604807 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:54.604784    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-29qkm"]
Jul 19 08:35:55.464275 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464242    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:35:55.464674 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464656    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.464797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464677    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.464797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464714    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.464797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464724    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.464797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464737    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.464797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.465051 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464852    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.465051 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464865    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.465051 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.464875    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" containerName="revoke-keys"
Jul 19 08:35:55.467979 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.467959    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:35:55.473270 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.473249    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:35:55.513604 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.513573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-m247k\" (UID: \"d2d75129-1e43-4700-a923-a5d81d2092ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:35:55.614266 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.614236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-m247k\" (UID: \"d2d75129-1e43-4700-a923-a5d81d2092ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:35:55.615016 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.614994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-m247k\" (UID: \"d2d75129-1e43-4700-a923-a5d81d2092ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:35:55.779630 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:55.779534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:35:56.115817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.115788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:35:56.455635 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.455610    2559 scope.go:117] "RemoveContainer" containerID="b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a"
Jul 19 08:35:56.462401 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.462374    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4" path="/var/lib/kubelet/pods/d78b2a6a-cb3a-4fa2-a0e4-f2f462311ed4/volumes"
Jul 19 08:35:56.597149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.597113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerStarted","Data":"fbebc553afff9c133d5c0ccfdc602a813ed5cc94663e13efdeeb987a8334b8b1"}
Jul 19 08:35:56.597318 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.597177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerStarted","Data":"78db8908d8b14e157c320cdea452eb235105eba14993975b6a92839f27004ed8"}
Jul 19 08:35:56.613028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:56.612981    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" podStartSLOduration=1.612966756 podStartE2EDuration="1.612966756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:35:56.609893603 +0000 UTC m=+3193.119491501" watchObservedRunningTime="2026-07-19 08:35:56.612966756 +0000 UTC m=+3193.122564655"
Jul 19 08:35:57.604130 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.604091    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerID="fbebc553afff9c133d5c0ccfdc602a813ed5cc94663e13efdeeb987a8334b8b1" exitCode=6
Jul 19 08:35:57.604559 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.604180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerDied","Data":"fbebc553afff9c133d5c0ccfdc602a813ed5cc94663e13efdeeb987a8334b8b1"}
Jul 19 08:35:57.604559 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.604367    2559 scope.go:117] "RemoveContainer" containerID="fbebc553afff9c133d5c0ccfdc602a813ed5cc94663e13efdeeb987a8334b8b1"
Jul 19 08:35:57.606134 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.606115    2559 generic.go:356] "Generic (PLEG): container finished" podID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerID="afa9ee166cc935d1f9ff3684839a840ebb80d2a0ad84bf24d910e81422c82304" exitCode=6
Jul 19 08:35:57.606226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.606189    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerDied","Data":"afa9ee166cc935d1f9ff3684839a840ebb80d2a0ad84bf24d910e81422c82304"}
Jul 19 08:35:57.606269 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.606243    2559 scope.go:117] "RemoveContainer" containerID="b6da86b5997b0fae638bd1a04f67d6d45735d4163ea063f7a189cd9ba306247a"
Jul 19 08:35:57.606610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:57.606495    2559 scope.go:117] "RemoveContainer" containerID="afa9ee166cc935d1f9ff3684839a840ebb80d2a0ad84bf24d910e81422c82304"
Jul 19 08:35:57.606714 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:57.606680    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z_odh-ai-gateway-infra(8104b6c9-880d-4225-82bc-6d132208d7ee)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee"
Jul 19 08:35:58.613461 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.613427    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerID="16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259" exitCode=6
Jul 19 08:35:58.613964 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.613514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerDied","Data":"16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259"}
Jul 19 08:35:58.613964 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.613583    2559 scope.go:117] "RemoveContainer" containerID="fbebc553afff9c133d5c0ccfdc602a813ed5cc94663e13efdeeb987a8334b8b1"
Jul 19 08:35:58.613964 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.613717    2559 scope.go:117] "RemoveContainer" containerID="16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259"
Jul 19 08:35:58.613964 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:58.613943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-m247k_odh-ai-gateway-infra(d2d75129-1e43-4700-a923-a5d81d2092ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae"
Jul 19 08:35:58.643497 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.643473    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:35:58.775224 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.775199    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:58.846206 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.846178    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca\") pod \"8104b6c9-880d-4225-82bc-6d132208d7ee\" (UID: \"8104b6c9-880d-4225-82bc-6d132208d7ee\") "
Jul 19 08:35:58.846516 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.846495    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8104b6c9-880d-4225-82bc-6d132208d7ee" (UID: "8104b6c9-880d-4225-82bc-6d132208d7ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:35:58.947394 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:58.947298    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8104b6c9-880d-4225-82bc-6d132208d7ee-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:35:59.621784 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.621751    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"
Jul 19 08:35:59.622259 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.621793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z" event={"ID":"8104b6c9-880d-4225-82bc-6d132208d7ee","Type":"ContainerDied","Data":"2e30403b89021f3b46dfd863e2c64a9783a5b982bbe95025f45e4a8bf852809b"}
Jul 19 08:35:59.622259 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.621848    2559 scope.go:117] "RemoveContainer" containerID="afa9ee166cc935d1f9ff3684839a840ebb80d2a0ad84bf24d910e81422c82304"
Jul 19 08:35:59.623882 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.623862    2559 scope.go:117] "RemoveContainer" containerID="16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259"
Jul 19 08:35:59.624141 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:35:59.624113    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-m247k_odh-ai-gateway-infra(d2d75129-1e43-4700-a923-a5d81d2092ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae"
Jul 19 08:35:59.655077 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.655052    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:35:59.662211 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:35:59.662184    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-dj67z"]
Jul 19 08:36:00.461753 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:00.461721    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" path="/var/lib/kubelet/pods/8104b6c9-880d-4225-82bc-6d132208d7ee/volumes"
Jul 19 08:36:12.454929 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:12.454895    2559 scope.go:117] "RemoveContainer" containerID="16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259"
Jul 19 08:36:13.700376 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:13.700344    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerID="fe6d6e767883ef1ebae90a4822a84dab38fa7de2fdc02d8e611ba7ae617bef12" exitCode=6
Jul 19 08:36:13.700830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:13.700419    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerDied","Data":"fe6d6e767883ef1ebae90a4822a84dab38fa7de2fdc02d8e611ba7ae617bef12"}
Jul 19 08:36:13.700830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:13.700476    2559 scope.go:117] "RemoveContainer" containerID="16517058915b7721a8151c56bd806783a30035174986759b79e29382a25ce259"
Jul 19 08:36:13.700830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:13.700784    2559 scope.go:117] "RemoveContainer" containerID="fe6d6e767883ef1ebae90a4822a84dab38fa7de2fdc02d8e611ba7ae617bef12"
Jul 19 08:36:13.701013 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:13.700973    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-m247k_odh-ai-gateway-infra(d2d75129-1e43-4700-a923-a5d81d2092ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae"
Jul 19 08:36:14.726046 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:14.726010    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:36:14.861775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:14.861752    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:36:14.989264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:14.989173    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca\") pod \"d2d75129-1e43-4700-a923-a5d81d2092ae\" (UID: \"d2d75129-1e43-4700-a923-a5d81d2092ae\") "
Jul 19 08:36:14.989486 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:14.989452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2d75129-1e43-4700-a923-a5d81d2092ae" (UID: "d2d75129-1e43-4700-a923-a5d81d2092ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:36:15.090291 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.090262    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d75129-1e43-4700-a923-a5d81d2092ae-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:36:15.713735 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.713694    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"
Jul 19 08:36:15.713909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.713733    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k" event={"ID":"d2d75129-1e43-4700-a923-a5d81d2092ae","Type":"ContainerDied","Data":"78db8908d8b14e157c320cdea452eb235105eba14993975b6a92839f27004ed8"}
Jul 19 08:36:15.713909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.713796    2559 scope.go:117] "RemoveContainer" containerID="fe6d6e767883ef1ebae90a4822a84dab38fa7de2fdc02d8e611ba7ae617bef12"
Jul 19 08:36:15.737894 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.737871    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:36:15.746446 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:15.746421    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-m247k"]
Jul 19 08:36:16.461457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:16.461424    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" path="/var/lib/kubelet/pods/d2d75129-1e43-4700-a923-a5d81d2092ae/volumes"
Jul 19 08:36:17.805285 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805254    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805599    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805608    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805613    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805622    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805628    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805643    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805650    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805724    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805731    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805737    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805743    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805750    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:17.805988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.805756    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:17.810178 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.810161    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:17.812927 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.812902    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:36:17.813858 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.813835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:36:17.815433 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.815406    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:17.913541 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:17.913512    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-45nfn\" (UID: \"42eac0ab-b9df-4609-94cc-4e86554820a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:18.014664 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.014627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-45nfn\" (UID: \"42eac0ab-b9df-4609-94cc-4e86554820a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:18.015241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.015223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-45nfn\" (UID: \"42eac0ab-b9df-4609-94cc-4e86554820a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:18.122426 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.122396    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:18.251928 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.251900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:18.253562 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:36:18.253536    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42eac0ab_b9df_4609_94cc_4e86554820a5.slice/crio-4f101dffda637e5fba241bd21ad0855eae1c0f665bca619a0012a07e62c43b4a WatchSource:0}: Error finding container 4f101dffda637e5fba241bd21ad0855eae1c0f665bca619a0012a07e62c43b4a: Status 404 returned error can't find the container with id 4f101dffda637e5fba241bd21ad0855eae1c0f665bca619a0012a07e62c43b4a
Jul 19 08:36:18.738147 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.737914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerStarted","Data":"dd20a3f990cbf5f48e65acd52164d298072a18d2945b136d106849bfc811da3f"}
Jul 19 08:36:18.738147 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.738056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerStarted","Data":"4f101dffda637e5fba241bd21ad0855eae1c0f665bca619a0012a07e62c43b4a"}
Jul 19 08:36:18.753238 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:18.753186    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" podStartSLOduration=1.753171837 podStartE2EDuration="1.753171837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:36:18.751192258 +0000 UTC m=+3215.260790157" watchObservedRunningTime="2026-07-19 08:36:18.753171837 +0000 UTC m=+3215.262769735"
Jul 19 08:36:19.745261 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:19.745229    2559 generic.go:356] "Generic (PLEG): container finished" podID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerID="dd20a3f990cbf5f48e65acd52164d298072a18d2945b136d106849bfc811da3f" exitCode=6
Jul 19 08:36:19.745718 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:19.745276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerDied","Data":"dd20a3f990cbf5f48e65acd52164d298072a18d2945b136d106849bfc811da3f"}
Jul 19 08:36:19.745718 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:19.745479    2559 scope.go:117] "RemoveContainer" containerID="dd20a3f990cbf5f48e65acd52164d298072a18d2945b136d106849bfc811da3f"
Jul 19 08:36:20.752909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:20.752812    2559 generic.go:356] "Generic (PLEG): container finished" podID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerID="49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003" exitCode=6
Jul 19 08:36:20.752909 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:20.752895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerDied","Data":"49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003"}
Jul 19 08:36:20.753406 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:20.752945    2559 scope.go:117] "RemoveContainer" containerID="dd20a3f990cbf5f48e65acd52164d298072a18d2945b136d106849bfc811da3f"
Jul 19 08:36:20.753406 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:20.753112    2559 scope.go:117] "RemoveContainer" containerID="49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003"
Jul 19 08:36:20.753406 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:20.753339    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-45nfn_odh-ai-gateway-infra(42eac0ab-b9df-4609-94cc-4e86554820a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5"
Jul 19 08:36:21.760346 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:21.760313    2559 scope.go:117] "RemoveContainer" containerID="49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003"
Jul 19 08:36:21.760742 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:21.760541    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-45nfn_odh-ai-gateway-infra(42eac0ab-b9df-4609-94cc-4e86554820a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5"
Jul 19 08:36:30.568775 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.568738    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:30.569160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.569096    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:30.569160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.569107    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d75129-1e43-4700-a923-a5d81d2092ae" containerName="revoke-keys"
Jul 19 08:36:30.569160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.569134    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:30.569160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.569141    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8104b6c9-880d-4225-82bc-6d132208d7ee" containerName="revoke-keys"
Jul 19 08:36:30.571834 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.571818    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:30.579650 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.579248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:30.733513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.733419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2\" (UID: \"5b5d0068-d7e5-49ed-97f8-718e9b33f63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:30.834074 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.834040    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2\" (UID: \"5b5d0068-d7e5-49ed-97f8-718e9b33f63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:30.834687 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.834667    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2\" (UID: \"5b5d0068-d7e5-49ed-97f8-718e9b33f63b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:30.883609 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:30.883582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:31.015343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:31.015309    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:31.016988 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:36:31.016962    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5d0068_d7e5_49ed_97f8_718e9b33f63b.slice/crio-25b966a1b137c8cb8622d295debf812f3be2571861981fe1b08dd77712638589 WatchSource:0}: Error finding container 25b966a1b137c8cb8622d295debf812f3be2571861981fe1b08dd77712638589: Status 404 returned error can't find the container with id 25b966a1b137c8cb8622d295debf812f3be2571861981fe1b08dd77712638589
Jul 19 08:36:31.817941 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:31.817859    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerID="9660caf59818fcdc9f13177f642c6c88a23d3518f9b548455828fef0a7ccb674" exitCode=6
Jul 19 08:36:31.818304 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:31.817943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerDied","Data":"9660caf59818fcdc9f13177f642c6c88a23d3518f9b548455828fef0a7ccb674"}
Jul 19 08:36:31.818304 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:31.818008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerStarted","Data":"25b966a1b137c8cb8622d295debf812f3be2571861981fe1b08dd77712638589"}
Jul 19 08:36:31.818304 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:31.818110    2559 scope.go:117] "RemoveContainer" containerID="9660caf59818fcdc9f13177f642c6c88a23d3518f9b548455828fef0a7ccb674"
Jul 19 08:36:32.455180 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.455145    2559 scope.go:117] "RemoveContainer" containerID="49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003"
Jul 19 08:36:32.825304 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.825273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerStarted","Data":"dbb261ab3c5302c5890498d65743cdf7cdccdd14cbaa52f6b506127549b5efac"}
Jul 19 08:36:32.827017 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.826994    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerID="6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd" exitCode=6
Jul 19 08:36:32.827152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.827069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerDied","Data":"6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd"}
Jul 19 08:36:32.827152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.827126    2559 scope.go:117] "RemoveContainer" containerID="9660caf59818fcdc9f13177f642c6c88a23d3518f9b548455828fef0a7ccb674"
Jul 19 08:36:32.827341 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:32.827325    2559 scope.go:117] "RemoveContainer" containerID="6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd"
Jul 19 08:36:32.827538 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:32.827513    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2_odh-ai-gateway-infra(5b5d0068-d7e5-49ed-97f8-718e9b33f63b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b"
Jul 19 08:36:33.812117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.812083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:33.815447 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.815430    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:33.822098 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.822072    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:33.834643 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.834616    2559 generic.go:356] "Generic (PLEG): container finished" podID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerID="dbb261ab3c5302c5890498d65743cdf7cdccdd14cbaa52f6b506127549b5efac" exitCode=6
Jul 19 08:36:33.835053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.834689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerDied","Data":"dbb261ab3c5302c5890498d65743cdf7cdccdd14cbaa52f6b506127549b5efac"}
Jul 19 08:36:33.835053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.834790    2559 scope.go:117] "RemoveContainer" containerID="49f7d86feca4c98fe6cff98ecb78b204da6e50110a93c49aaca571d1eabb3003"
Jul 19 08:36:33.835175 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.835075    2559 scope.go:117] "RemoveContainer" containerID="dbb261ab3c5302c5890498d65743cdf7cdccdd14cbaa52f6b506127549b5efac"
Jul 19 08:36:33.835355 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:33.835328    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-45nfn_odh-ai-gateway-infra(42eac0ab-b9df-4609-94cc-4e86554820a5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5"
Jul 19 08:36:33.836655 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.836638    2559 scope.go:117] "RemoveContainer" containerID="6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd"
Jul 19 08:36:33.836895 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:33.836872    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2_odh-ai-gateway-infra(5b5d0068-d7e5-49ed-97f8-718e9b33f63b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b"
Jul 19 08:36:33.849711 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.849679    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:33.961550 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:33.961518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr\" (UID: \"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:34.061966 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.061934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr\" (UID: \"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:34.062584 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.062533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr\" (UID: \"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:34.127801 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.127771    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:34.264230 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.264202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:34.265154 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:36:34.265125    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fbbfdd_14af_44b8_b209_8c9a4d8f3d65.slice/crio-1d1247e29147a29b3326992cba5bdf58a190bfa6bcc4267ae1fbdcee26ab8bc3 WatchSource:0}: Error finding container 1d1247e29147a29b3326992cba5bdf58a190bfa6bcc4267ae1fbdcee26ab8bc3: Status 404 returned error can't find the container with id 1d1247e29147a29b3326992cba5bdf58a190bfa6bcc4267ae1fbdcee26ab8bc3
Jul 19 08:36:34.844074 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.844031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerStarted","Data":"42bcab7e478a48837e650ff037af2a651afee10bc591fd6cc6dec420a5954633"}
Jul 19 08:36:34.844513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.844115    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerStarted","Data":"1d1247e29147a29b3326992cba5bdf58a190bfa6bcc4267ae1fbdcee26ab8bc3"}
Jul 19 08:36:34.859954 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:34.859904    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" podStartSLOduration=1.859888796 podStartE2EDuration="1.859888796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:36:34.857883783 +0000 UTC m=+3231.367481683" watchObservedRunningTime="2026-07-19 08:36:34.859888796 +0000 UTC m=+3231.369486694"
Jul 19 08:36:35.027883 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.027853    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:35.068469 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.068435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca\") pod \"42eac0ab-b9df-4609-94cc-4e86554820a5\" (UID: \"42eac0ab-b9df-4609-94cc-4e86554820a5\") "
Jul 19 08:36:35.068770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.068747    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42eac0ab-b9df-4609-94cc-4e86554820a5" (UID: "42eac0ab-b9df-4609-94cc-4e86554820a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:36:35.169524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.169493    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42eac0ab-b9df-4609-94cc-4e86554820a5-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:36:35.852198 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.852166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"
Jul 19 08:36:35.852684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.852165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn" event={"ID":"42eac0ab-b9df-4609-94cc-4e86554820a5","Type":"ContainerDied","Data":"4f101dffda637e5fba241bd21ad0855eae1c0f665bca619a0012a07e62c43b4a"}
Jul 19 08:36:35.852684 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.852294    2559 scope.go:117] "RemoveContainer" containerID="dbb261ab3c5302c5890498d65743cdf7cdccdd14cbaa52f6b506127549b5efac"
Jul 19 08:36:35.853774 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.853747    2559 generic.go:356] "Generic (PLEG): container finished" podID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerID="42bcab7e478a48837e650ff037af2a651afee10bc591fd6cc6dec420a5954633" exitCode=6
Jul 19 08:36:35.853844 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.853819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerDied","Data":"42bcab7e478a48837e650ff037af2a651afee10bc591fd6cc6dec420a5954633"}
Jul 19 08:36:35.853984 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.853969    2559 scope.go:117] "RemoveContainer" containerID="42bcab7e478a48837e650ff037af2a651afee10bc591fd6cc6dec420a5954633"
Jul 19 08:36:35.884205 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.884179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:35.887573 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:35.887551    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-45nfn"]
Jul 19 08:36:36.461604 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:36.461578    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" path="/var/lib/kubelet/pods/42eac0ab-b9df-4609-94cc-4e86554820a5/volumes"
Jul 19 08:36:36.862714 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:36.862678    2559 generic.go:356] "Generic (PLEG): container finished" podID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerID="7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9" exitCode=6
Jul 19 08:36:36.863159 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:36.862754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerDied","Data":"7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9"}
Jul 19 08:36:36.863159 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:36.862815    2559 scope.go:117] "RemoveContainer" containerID="42bcab7e478a48837e650ff037af2a651afee10bc591fd6cc6dec420a5954633"
Jul 19 08:36:36.863159 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:36.862989    2559 scope.go:117] "RemoveContainer" containerID="7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9"
Jul 19 08:36:36.863278 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:36.863247    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr_odh-ai-gateway-infra(50fbbfdd-14af-44b8-b209-8c9a4d8f3d65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65"
Jul 19 08:36:37.870006 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:37.869979    2559 scope.go:117] "RemoveContainer" containerID="7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9"
Jul 19 08:36:37.870404 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:37.870194    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr_odh-ai-gateway-infra(50fbbfdd-14af-44b8-b209-8c9a4d8f3d65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65"
Jul 19 08:36:44.461695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:44.461664    2559 scope.go:117] "RemoveContainer" containerID="6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd"
Jul 19 08:36:44.911272 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:44.911247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerStarted","Data":"ff5054d908a0e95aa8f0653d18d962d16d4c09999f986d2239885dee0838d761"}
Jul 19 08:36:44.926601 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:44.926559    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" podStartSLOduration=14.926542995 podStartE2EDuration="14.926542995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:36:44.925219108 +0000 UTC m=+3241.434817010" watchObservedRunningTime="2026-07-19 08:36:44.926542995 +0000 UTC m=+3241.436140924"
Jul 19 08:36:45.919061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:45.919031    2559 generic.go:356] "Generic (PLEG): container finished" podID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerID="ff5054d908a0e95aa8f0653d18d962d16d4c09999f986d2239885dee0838d761" exitCode=6
Jul 19 08:36:45.919490 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:45.919106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerDied","Data":"ff5054d908a0e95aa8f0653d18d962d16d4c09999f986d2239885dee0838d761"}
Jul 19 08:36:45.919490 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:45.919165    2559 scope.go:117] "RemoveContainer" containerID="6b5fe5c14638386748e09b9d3023ca4b952ce90482547fcb78c3c376445f0dbd"
Jul 19 08:36:45.919490 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:45.919456    2559 scope.go:117] "RemoveContainer" containerID="ff5054d908a0e95aa8f0653d18d962d16d4c09999f986d2239885dee0838d761"
Jul 19 08:36:45.919713 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:45.919681    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2_odh-ai-gateway-infra(5b5d0068-d7e5-49ed-97f8-718e9b33f63b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b"
Jul 19 08:36:45.935155 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:45.935132    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:47.060610 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.060582    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:47.184268 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.184181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca\") pod \"5b5d0068-d7e5-49ed-97f8-718e9b33f63b\" (UID: \"5b5d0068-d7e5-49ed-97f8-718e9b33f63b\") "
Jul 19 08:36:47.184552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.184530    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b5d0068-d7e5-49ed-97f8-718e9b33f63b" (UID: "5b5d0068-d7e5-49ed-97f8-718e9b33f63b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:36:47.285525 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.285492    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5d0068-d7e5-49ed-97f8-718e9b33f63b-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:36:47.940647 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.940605    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"
Jul 19 08:36:47.940647 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.940616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2" event={"ID":"5b5d0068-d7e5-49ed-97f8-718e9b33f63b","Type":"ContainerDied","Data":"25b966a1b137c8cb8622d295debf812f3be2571861981fe1b08dd77712638589"}
Jul 19 08:36:47.940892 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.940754    2559 scope.go:117] "RemoveContainer" containerID="ff5054d908a0e95aa8f0653d18d962d16d4c09999f986d2239885dee0838d761"
Jul 19 08:36:47.964177 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.964148    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:47.967837 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:47.967815    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-fxzl2"]
Jul 19 08:36:48.461327 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:48.461293    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" path="/var/lib/kubelet/pods/5b5d0068-d7e5-49ed-97f8-718e9b33f63b/volumes"
Jul 19 08:36:50.455151 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:50.455116    2559 scope.go:117] "RemoveContainer" containerID="7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9"
Jul 19 08:36:50.960291 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:50.960259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerStarted","Data":"ab1b2a323e3557c6c3b5b5bd4f9f599eca7f64f6ee4eff04918bd81a87ac25e9"}
Jul 19 08:36:51.967384 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:51.967354    2559 generic.go:356] "Generic (PLEG): container finished" podID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerID="ab1b2a323e3557c6c3b5b5bd4f9f599eca7f64f6ee4eff04918bd81a87ac25e9" exitCode=6
Jul 19 08:36:51.967856 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:51.967432    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerDied","Data":"ab1b2a323e3557c6c3b5b5bd4f9f599eca7f64f6ee4eff04918bd81a87ac25e9"}
Jul 19 08:36:51.967856 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:51.967495    2559 scope.go:117] "RemoveContainer" containerID="7849264caac2b8314b52a5c1b2ba34e16ea5834a52bbd881aed6d1a98c496ee9"
Jul 19 08:36:51.967856 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:51.967787    2559 scope.go:117] "RemoveContainer" containerID="ab1b2a323e3557c6c3b5b5bd4f9f599eca7f64f6ee4eff04918bd81a87ac25e9"
Jul 19 08:36:51.968043 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:51.968022    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr_odh-ai-gateway-infra(50fbbfdd-14af-44b8-b209-8c9a4d8f3d65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65"
Jul 19 08:36:51.990507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:51.990483    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:53.108172 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.108145    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:53.238756 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.238643    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca\") pod \"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65\" (UID: \"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65\") "
Jul 19 08:36:53.238991 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.238968    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" (UID: "50fbbfdd-14af-44b8-b209-8c9a4d8f3d65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:36:53.340133 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.340099    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:36:53.980213 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.980182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"
Jul 19 08:36:53.980426 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.980221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr" event={"ID":"50fbbfdd-14af-44b8-b209-8c9a4d8f3d65","Type":"ContainerDied","Data":"1d1247e29147a29b3326992cba5bdf58a190bfa6bcc4267ae1fbdcee26ab8bc3"}
Jul 19 08:36:53.980426 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:53.980279    2559 scope.go:117] "RemoveContainer" containerID="ab1b2a323e3557c6c3b5b5bd4f9f599eca7f64f6ee4eff04918bd81a87ac25e9"
Jul 19 08:36:54.006710 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:54.006675    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:54.010166 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:54.010144    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-flhxr"]
Jul 19 08:36:54.460418 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:54.460387    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" path="/var/lib/kubelet/pods/50fbbfdd-14af-44b8-b209-8c9a4d8f3d65/volumes"
Jul 19 08:36:55.510286 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510249    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510585    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510595    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510603    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510609    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510619    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510637    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510650    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510656    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510663    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510668    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510675    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.510672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510680    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510688    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510693    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510767    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510774    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510779    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510784    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:36:55.511121 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.510790    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:36:55.513485 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.513469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:36:55.515760 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.515739    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:36:55.515875 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.515743    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:36:55.520496 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.520470    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:36:55.661107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.661069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6\" (UID: \"132d2f9f-4cd8-404f-bed8-838d31c9eaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:36:55.762383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.762291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6\" (UID: \"132d2f9f-4cd8-404f-bed8-838d31c9eaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:36:55.762999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.762973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6\" (UID: \"132d2f9f-4cd8-404f-bed8-838d31c9eaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:36:55.826423 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.826382    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:36:55.959345 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.959310    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:36:55.961917 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:36:55.961885    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132d2f9f_4cd8_404f_bed8_838d31c9eaa0.slice/crio-318a6ba680a3700e3805817db61f5b7d16fd0da2c9710357c7c1e1d5104af1aa WatchSource:0}: Error finding container 318a6ba680a3700e3805817db61f5b7d16fd0da2c9710357c7c1e1d5104af1aa: Status 404 returned error can't find the container with id 318a6ba680a3700e3805817db61f5b7d16fd0da2c9710357c7c1e1d5104af1aa
Jul 19 08:36:55.993634 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:55.993608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" event={"ID":"132d2f9f-4cd8-404f-bed8-838d31c9eaa0","Type":"ContainerStarted","Data":"318a6ba680a3700e3805817db61f5b7d16fd0da2c9710357c7c1e1d5104af1aa"}
Jul 19 08:36:57.001212 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:57.001121    2559 generic.go:356] "Generic (PLEG): container finished" podID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerID="0d42c590e4c88411230839c0b1aace126282a46ca77fc3c9bbc164b3a468081c" exitCode=6
Jul 19 08:36:57.001583 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:57.001211    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" event={"ID":"132d2f9f-4cd8-404f-bed8-838d31c9eaa0","Type":"ContainerDied","Data":"0d42c590e4c88411230839c0b1aace126282a46ca77fc3c9bbc164b3a468081c"}
Jul 19 08:36:57.001583 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:57.001378    2559 scope.go:117] "RemoveContainer" containerID="0d42c590e4c88411230839c0b1aace126282a46ca77fc3c9bbc164b3a468081c"
Jul 19 08:36:58.009856 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:58.009759    2559 generic.go:356] "Generic (PLEG): container finished" podID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerID="95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007" exitCode=6
Jul 19 08:36:58.009856 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:58.009841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" event={"ID":"132d2f9f-4cd8-404f-bed8-838d31c9eaa0","Type":"ContainerDied","Data":"95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007"}
Jul 19 08:36:58.010363 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:58.009906    2559 scope.go:117] "RemoveContainer" containerID="0d42c590e4c88411230839c0b1aace126282a46ca77fc3c9bbc164b3a468081c"
Jul 19 08:36:58.010363 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:58.010130    2559 scope.go:117] "RemoveContainer" containerID="95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007"
Jul 19 08:36:58.010363 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:58.010352    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6_odh-ai-gateway-infra(132d2f9f-4cd8-404f-bed8-838d31c9eaa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0"
Jul 19 08:36:59.017215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:36:59.017183    2559 scope.go:117] "RemoveContainer" containerID="95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007"
Jul 19 08:36:59.017619 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:36:59.017403    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6_odh-ai-gateway-infra(132d2f9f-4cd8-404f-bed8-838d31c9eaa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0"
Jul 19 08:37:07.846406 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846370    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:07.846852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846757    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:37:07.846852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846768    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:37:07.846852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846828    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42eac0ab-b9df-4609-94cc-4e86554820a5" containerName="revoke-keys"
Jul 19 08:37:07.846852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846836    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b5d0068-d7e5-49ed-97f8-718e9b33f63b" containerName="revoke-keys"
Jul 19 08:37:07.846852 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.846844    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fbbfdd-14af-44b8-b209-8c9a4d8f3d65" containerName="revoke-keys"
Jul 19 08:37:07.851137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.851121    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:07.856050 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.856021    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:07.972992 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:07.972960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6\" (UID: \"2325bd8b-8530-4545-a6d0-c6bdc6d717dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:08.074075 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:08.074046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6\" (UID: \"2325bd8b-8530-4545-a6d0-c6bdc6d717dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:08.074665 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:08.074647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6\" (UID: \"2325bd8b-8530-4545-a6d0-c6bdc6d717dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:08.163479 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:08.163409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:08.306211 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:08.306179    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:08.307730 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:37:08.307687    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2325bd8b_8530_4545_a6d0_c6bdc6d717dd.slice/crio-0fb5eedae0783a23d434dd4b2ec7dc4def96b9a1a46fd750c9bc45e8294ce627 WatchSource:0}: Error finding container 0fb5eedae0783a23d434dd4b2ec7dc4def96b9a1a46fd750c9bc45e8294ce627: Status 404 returned error can't find the container with id 0fb5eedae0783a23d434dd4b2ec7dc4def96b9a1a46fd750c9bc45e8294ce627
Jul 19 08:37:09.073413 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:09.073373    2559 generic.go:356] "Generic (PLEG): container finished" podID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerID="b5d4a393f4011bf16d26abb22a4e942a9b839578ee795394255bca9d9ca895fa" exitCode=6
Jul 19 08:37:09.073789 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:09.073457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" event={"ID":"2325bd8b-8530-4545-a6d0-c6bdc6d717dd","Type":"ContainerDied","Data":"b5d4a393f4011bf16d26abb22a4e942a9b839578ee795394255bca9d9ca895fa"}
Jul 19 08:37:09.073789 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:09.073523    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" event={"ID":"2325bd8b-8530-4545-a6d0-c6bdc6d717dd","Type":"ContainerStarted","Data":"0fb5eedae0783a23d434dd4b2ec7dc4def96b9a1a46fd750c9bc45e8294ce627"}
Jul 19 08:37:09.073789 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:09.073655    2559 scope.go:117] "RemoveContainer" containerID="b5d4a393f4011bf16d26abb22a4e942a9b839578ee795394255bca9d9ca895fa"
Jul 19 08:37:10.081152 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:10.081122    2559 generic.go:356] "Generic (PLEG): container finished" podID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerID="b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992" exitCode=6
Jul 19 08:37:10.081560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:10.081214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" event={"ID":"2325bd8b-8530-4545-a6d0-c6bdc6d717dd","Type":"ContainerDied","Data":"b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992"}
Jul 19 08:37:10.081560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:10.081274    2559 scope.go:117] "RemoveContainer" containerID="b5d4a393f4011bf16d26abb22a4e942a9b839578ee795394255bca9d9ca895fa"
Jul 19 08:37:10.081560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:10.081459    2559 scope.go:117] "RemoveContainer" containerID="b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992"
Jul 19 08:37:10.081760 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:10.081739    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6_odh-ai-gateway-infra(2325bd8b-8530-4545-a6d0-c6bdc6d717dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd"
Jul 19 08:37:10.461208 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:10.461126    2559 scope.go:117] "RemoveContainer" containerID="95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007"
Jul 19 08:37:11.089215 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:11.089189    2559 scope.go:117] "RemoveContainer" containerID="b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992"
Jul 19 08:37:11.089603 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:11.089473    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6_odh-ai-gateway-infra(2325bd8b-8530-4545-a6d0-c6bdc6d717dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd"
Jul 19 08:37:11.090738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:11.090691    2559 generic.go:356] "Generic (PLEG): container finished" podID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerID="264d478c833e12d1842e3d441022c3a8abcfc5f371bfb2fcb5de2ff3a626fe84" exitCode=6
Jul 19 08:37:11.090738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:11.090726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" event={"ID":"132d2f9f-4cd8-404f-bed8-838d31c9eaa0","Type":"ContainerDied","Data":"264d478c833e12d1842e3d441022c3a8abcfc5f371bfb2fcb5de2ff3a626fe84"}
Jul 19 08:37:11.090895 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:11.090808    2559 scope.go:117] "RemoveContainer" containerID="95990fd4618994b624ab5e41643ff2980220e5520c5db2abb1dbcd82440a2007"
Jul 19 08:37:11.091079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:11.091062    2559 scope.go:117] "RemoveContainer" containerID="264d478c833e12d1842e3d441022c3a8abcfc5f371bfb2fcb5de2ff3a626fe84"
Jul 19 08:37:11.091287 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:11.091268    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6_odh-ai-gateway-infra(132d2f9f-4cd8-404f-bed8-838d31c9eaa0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0"
Jul 19 08:37:12.128062 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:12.128025    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:37:12.258826 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:12.258801    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:37:12.412315 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:12.412231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca\") pod \"132d2f9f-4cd8-404f-bed8-838d31c9eaa0\" (UID: \"132d2f9f-4cd8-404f-bed8-838d31c9eaa0\") "
Jul 19 08:37:12.412589 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:12.412568    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "132d2f9f-4cd8-404f-bed8-838d31c9eaa0" (UID: "132d2f9f-4cd8-404f-bed8-838d31c9eaa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:37:12.512987 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:12.512956    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/132d2f9f-4cd8-404f-bed8-838d31c9eaa0-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:37:13.106729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:13.105394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6" event={"ID":"132d2f9f-4cd8-404f-bed8-838d31c9eaa0","Type":"ContainerDied","Data":"318a6ba680a3700e3805817db61f5b7d16fd0da2c9710357c7c1e1d5104af1aa"}
Jul 19 08:37:13.106729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:13.105464    2559 scope.go:117] "RemoveContainer" containerID="264d478c833e12d1842e3d441022c3a8abcfc5f371bfb2fcb5de2ff3a626fe84"
Jul 19 08:37:13.106729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:13.105568    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"
Jul 19 08:37:13.121961 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:13.121935    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:37:13.125562 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:13.125538    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-55pb6"]
Jul 19 08:37:14.460721 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:14.460670    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" path="/var/lib/kubelet/pods/132d2f9f-4cd8-404f-bed8-838d31c9eaa0/volumes"
Jul 19 08:37:20.610847 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.610811    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611171    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611182    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611204    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611209    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611267    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.611310 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.611274    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:20.615617 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.615600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:20.621454 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.621428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:20.682797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.682753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z\" (UID: \"0cf94e94-7090-4a26-8a16-5f15a60587e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:20.784300 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.784266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z\" (UID: \"0cf94e94-7090-4a26-8a16-5f15a60587e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:20.784903 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.784885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z\" (UID: \"0cf94e94-7090-4a26-8a16-5f15a60587e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:20.927423 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:20.927310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:21.059863 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:21.059842    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:21.060684 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:37:21.060657    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf94e94_7090_4a26_8a16_5f15a60587e9.slice/crio-bfc365dc730debba4ea5355d24983af96bc226d5457af6a15d8b785edd7ca229 WatchSource:0}: Error finding container bfc365dc730debba4ea5355d24983af96bc226d5457af6a15d8b785edd7ca229: Status 404 returned error can't find the container with id bfc365dc730debba4ea5355d24983af96bc226d5457af6a15d8b785edd7ca229
Jul 19 08:37:21.150908 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:21.150876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" event={"ID":"0cf94e94-7090-4a26-8a16-5f15a60587e9","Type":"ContainerStarted","Data":"bfc365dc730debba4ea5355d24983af96bc226d5457af6a15d8b785edd7ca229"}
Jul 19 08:37:22.157545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:22.157512    2559 generic.go:356] "Generic (PLEG): container finished" podID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerID="c4456170f6ad93429cb2c4e4b53bb42b081eec1149461489f7baa9bad84f2278" exitCode=6
Jul 19 08:37:22.158008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:22.157606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" event={"ID":"0cf94e94-7090-4a26-8a16-5f15a60587e9","Type":"ContainerDied","Data":"c4456170f6ad93429cb2c4e4b53bb42b081eec1149461489f7baa9bad84f2278"}
Jul 19 08:37:22.158160 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:22.158132    2559 scope.go:117] "RemoveContainer" containerID="c4456170f6ad93429cb2c4e4b53bb42b081eec1149461489f7baa9bad84f2278"
Jul 19 08:37:23.165283 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:23.165193    2559 generic.go:356] "Generic (PLEG): container finished" podID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerID="5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce" exitCode=6
Jul 19 08:37:23.165672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:23.165272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" event={"ID":"0cf94e94-7090-4a26-8a16-5f15a60587e9","Type":"ContainerDied","Data":"5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce"}
Jul 19 08:37:23.165672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:23.165342    2559 scope.go:117] "RemoveContainer" containerID="c4456170f6ad93429cb2c4e4b53bb42b081eec1149461489f7baa9bad84f2278"
Jul 19 08:37:23.165672 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:23.165491    2559 scope.go:117] "RemoveContainer" containerID="5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce"
Jul 19 08:37:23.165883 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:23.165754    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z_odh-ai-gateway-infra(0cf94e94-7090-4a26-8a16-5f15a60587e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9"
Jul 19 08:37:24.172390 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:24.172368    2559 scope.go:117] "RemoveContainer" containerID="5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce"
Jul 19 08:37:24.172793 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:24.172561    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z_odh-ai-gateway-infra(0cf94e94-7090-4a26-8a16-5f15a60587e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9"
Jul 19 08:37:26.455242 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:26.455212    2559 scope.go:117] "RemoveContainer" containerID="b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992"
Jul 19 08:37:27.190765 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:27.190732    2559 generic.go:356] "Generic (PLEG): container finished" podID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerID="70727a5dee43b8d4675c069c4fd809dab575103cd6f8d29f694809ca903603c1" exitCode=6
Jul 19 08:37:27.190947 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:27.190804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" event={"ID":"2325bd8b-8530-4545-a6d0-c6bdc6d717dd","Type":"ContainerDied","Data":"70727a5dee43b8d4675c069c4fd809dab575103cd6f8d29f694809ca903603c1"}
Jul 19 08:37:27.190947 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:27.190863    2559 scope.go:117] "RemoveContainer" containerID="b093db091ce0b5f7e55a4da83119aeaa1583f77bd36d5a7d35e75e06ac7a4992"
Jul 19 08:37:27.191171 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:27.191147    2559 scope.go:117] "RemoveContainer" containerID="70727a5dee43b8d4675c069c4fd809dab575103cd6f8d29f694809ca903603c1"
Jul 19 08:37:27.191384 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:27.191365    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6_odh-ai-gateway-infra(2325bd8b-8530-4545-a6d0-c6bdc6d717dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd"
Jul 19 08:37:28.215130 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:28.215093    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:28.346246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:28.346225    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:28.457806 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:28.457773    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca\") pod \"2325bd8b-8530-4545-a6d0-c6bdc6d717dd\" (UID: \"2325bd8b-8530-4545-a6d0-c6bdc6d717dd\") "
Jul 19 08:37:28.458114 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:28.458093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2325bd8b-8530-4545-a6d0-c6bdc6d717dd" (UID: "2325bd8b-8530-4545-a6d0-c6bdc6d717dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:37:28.558485 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:28.558452    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2325bd8b-8530-4545-a6d0-c6bdc6d717dd-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:37:29.204385 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:29.204356    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"
Jul 19 08:37:29.204592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:29.204395    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6" event={"ID":"2325bd8b-8530-4545-a6d0-c6bdc6d717dd","Type":"ContainerDied","Data":"0fb5eedae0783a23d434dd4b2ec7dc4def96b9a1a46fd750c9bc45e8294ce627"}
Jul 19 08:37:29.204592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:29.204458    2559 scope.go:117] "RemoveContainer" containerID="70727a5dee43b8d4675c069c4fd809dab575103cd6f8d29f694809ca903603c1"
Jul 19 08:37:29.220549 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:29.220527    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:29.225554 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:29.225533    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-n6qz6"]
Jul 19 08:37:30.461042 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:30.461012    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" path="/var/lib/kubelet/pods/2325bd8b-8530-4545-a6d0-c6bdc6d717dd/volumes"
Jul 19 08:37:33.855335 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855296    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855651    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855662    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855669    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855675    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855688    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855694    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:33.855769 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855769    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855998 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855776    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="132d2f9f-4cd8-404f-bed8-838d31c9eaa0" containerName="revoke-keys"
Jul 19 08:37:33.855998 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855785    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.855998 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.855792    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:33.860280 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.860260    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:33.865110 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.865084    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:33.907776 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:33.907746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv\" (UID: \"a84337bf-ff79-4ea3-913d-ee86c9afa408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:34.009096 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:34.009062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv\" (UID: \"a84337bf-ff79-4ea3-913d-ee86c9afa408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:34.009679 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:34.009660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv\" (UID: \"a84337bf-ff79-4ea3-913d-ee86c9afa408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:34.172416 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:34.172336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:34.303086 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:34.303056    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:34.304513 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:37:34.304489    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84337bf_ff79_4ea3_913d_ee86c9afa408.slice/crio-33ce750c2273832f03a50dfecff04fb59d044a0c7c42c07d500acbcaf76eeb1b WatchSource:0}: Error finding container 33ce750c2273832f03a50dfecff04fb59d044a0c7c42c07d500acbcaf76eeb1b: Status 404 returned error can't find the container with id 33ce750c2273832f03a50dfecff04fb59d044a0c7c42c07d500acbcaf76eeb1b
Jul 19 08:37:35.240722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:35.240603    2559 generic.go:356] "Generic (PLEG): container finished" podID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerID="3ffd6e0cfd02bac7dd3bfe5ba43235571713b85e65ba79146f3a6df264219c1b" exitCode=6
Jul 19 08:37:35.241111 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:35.240728    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" event={"ID":"a84337bf-ff79-4ea3-913d-ee86c9afa408","Type":"ContainerDied","Data":"3ffd6e0cfd02bac7dd3bfe5ba43235571713b85e65ba79146f3a6df264219c1b"}
Jul 19 08:37:35.241111 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:35.240800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" event={"ID":"a84337bf-ff79-4ea3-913d-ee86c9afa408","Type":"ContainerStarted","Data":"33ce750c2273832f03a50dfecff04fb59d044a0c7c42c07d500acbcaf76eeb1b"}
Jul 19 08:37:35.241111 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:35.240925    2559 scope.go:117] "RemoveContainer" containerID="3ffd6e0cfd02bac7dd3bfe5ba43235571713b85e65ba79146f3a6df264219c1b"
Jul 19 08:37:35.455017 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:35.454988    2559 scope.go:117] "RemoveContainer" containerID="5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce"
Jul 19 08:37:36.248080 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.247999    2559 generic.go:356] "Generic (PLEG): container finished" podID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerID="4b2fa7e7d3340481d045e73123ea98bcd8feb57822d0d8e5ea970d81b286ebbb" exitCode=6
Jul 19 08:37:36.248532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.248072    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" event={"ID":"0cf94e94-7090-4a26-8a16-5f15a60587e9","Type":"ContainerDied","Data":"4b2fa7e7d3340481d045e73123ea98bcd8feb57822d0d8e5ea970d81b286ebbb"}
Jul 19 08:37:36.248532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.248146    2559 scope.go:117] "RemoveContainer" containerID="5af41aab9696f0af62381a11766eab4393d702298b6913591212a002203852ce"
Jul 19 08:37:36.248532 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.248419    2559 scope.go:117] "RemoveContainer" containerID="4b2fa7e7d3340481d045e73123ea98bcd8feb57822d0d8e5ea970d81b286ebbb"
Jul 19 08:37:36.248731 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:36.248640    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z_odh-ai-gateway-infra(0cf94e94-7090-4a26-8a16-5f15a60587e9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9"
Jul 19 08:37:36.250112 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.250089    2559 generic.go:356] "Generic (PLEG): container finished" podID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerID="4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1" exitCode=6
Jul 19 08:37:36.250249 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.250175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" event={"ID":"a84337bf-ff79-4ea3-913d-ee86c9afa408","Type":"ContainerDied","Data":"4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1"}
Jul 19 08:37:36.250343 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.250331    2559 scope.go:117] "RemoveContainer" containerID="4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1"
Jul 19 08:37:36.250491 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:36.250476    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv_odh-ai-gateway-infra(a84337bf-ff79-4ea3-913d-ee86c9afa408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408"
Jul 19 08:37:36.259371 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:36.259353    2559 scope.go:117] "RemoveContainer" containerID="3ffd6e0cfd02bac7dd3bfe5ba43235571713b85e65ba79146f3a6df264219c1b"
Jul 19 08:37:37.258411 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.258384    2559 scope.go:117] "RemoveContainer" containerID="4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1"
Jul 19 08:37:37.258795 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:37.258575    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv_odh-ai-gateway-infra(a84337bf-ff79-4ea3-913d-ee86c9afa408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408"
Jul 19 08:37:37.273178 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.273144    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:37.413459 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.413428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:37.540268 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.540165    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca\") pod \"0cf94e94-7090-4a26-8a16-5f15a60587e9\" (UID: \"0cf94e94-7090-4a26-8a16-5f15a60587e9\") "
Jul 19 08:37:37.540738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.540544    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cf94e94-7090-4a26-8a16-5f15a60587e9" (UID: "0cf94e94-7090-4a26-8a16-5f15a60587e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:37:37.540738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:37.540678    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf94e94-7090-4a26-8a16-5f15a60587e9-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:37:38.264560 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.264534    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"
Jul 19 08:37:38.265041 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.264536    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z" event={"ID":"0cf94e94-7090-4a26-8a16-5f15a60587e9","Type":"ContainerDied","Data":"bfc365dc730debba4ea5355d24983af96bc226d5457af6a15d8b785edd7ca229"}
Jul 19 08:37:38.265041 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.264670    2559 scope.go:117] "RemoveContainer" containerID="4b2fa7e7d3340481d045e73123ea98bcd8feb57822d0d8e5ea970d81b286ebbb"
Jul 19 08:37:38.291581 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.291556    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:38.295190 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.295167    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-6q44z"]
Jul 19 08:37:38.463740 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:38.462388    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" path="/var/lib/kubelet/pods/0cf94e94-7090-4a26-8a16-5f15a60587e9/volumes"
Jul 19 08:37:45.550985 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.550948    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551316    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551335    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551342    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551348    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551366    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:45.551375 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551371    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2325bd8b-8530-4545-a6d0-c6bdc6d717dd" containerName="revoke-keys"
Jul 19 08:37:45.551575 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551430    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551575 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551438    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.551575 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.551444    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:37:45.556072 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.556056    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:37:45.560530 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.560497    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:37:45.709571 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.709536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl\" (UID: \"6b0240d5-ad57-4a49-8a88-4ee279c4d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:37:45.810822 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.810795    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl\" (UID: \"6b0240d5-ad57-4a49-8a88-4ee279c4d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:37:45.811420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.811402    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl\" (UID: \"6b0240d5-ad57-4a49-8a88-4ee279c4d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:37:45.867879 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.867830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:37:45.999123 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:45.999093    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:37:46.000779 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:37:46.000753    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0240d5_ad57_4a49_8a88_4ee279c4d872.slice/crio-5b98ad551115f2ad8a0c10cdebf2981c036f4cf9e779eafc612cf7863da1a1ac WatchSource:0}: Error finding container 5b98ad551115f2ad8a0c10cdebf2981c036f4cf9e779eafc612cf7863da1a1ac: Status 404 returned error can't find the container with id 5b98ad551115f2ad8a0c10cdebf2981c036f4cf9e779eafc612cf7863da1a1ac
Jul 19 08:37:46.312298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:46.312264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerStarted","Data":"c9c87b4f9d4969bdc0c08a7715b7ed23a30db67acff60f79031f0d860badf1fc"}
Jul 19 08:37:46.312443 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:46.312314    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerStarted","Data":"5b98ad551115f2ad8a0c10cdebf2981c036f4cf9e779eafc612cf7863da1a1ac"}
Jul 19 08:37:46.328226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:46.328178    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" podStartSLOduration=1.328163039 podStartE2EDuration="1.328163039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:37:46.326213934 +0000 UTC m=+3302.835811845" watchObservedRunningTime="2026-07-19 08:37:46.328163039 +0000 UTC m=+3302.837760938"
Jul 19 08:37:47.320220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:47.320179    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerID="c9c87b4f9d4969bdc0c08a7715b7ed23a30db67acff60f79031f0d860badf1fc" exitCode=6
Jul 19 08:37:47.320667 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:47.320264    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerDied","Data":"c9c87b4f9d4969bdc0c08a7715b7ed23a30db67acff60f79031f0d860badf1fc"}
Jul 19 08:37:47.320667 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:47.320520    2559 scope.go:117] "RemoveContainer" containerID="c9c87b4f9d4969bdc0c08a7715b7ed23a30db67acff60f79031f0d860badf1fc"
Jul 19 08:37:48.327436 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:48.327400    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerID="2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67" exitCode=6
Jul 19 08:37:48.327876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:48.327449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerDied","Data":"2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67"}
Jul 19 08:37:48.327876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:48.327506    2559 scope.go:117] "RemoveContainer" containerID="c9c87b4f9d4969bdc0c08a7715b7ed23a30db67acff60f79031f0d860badf1fc"
Jul 19 08:37:48.327876 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:48.327745    2559 scope.go:117] "RemoveContainer" containerID="2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67"
Jul 19 08:37:48.328016 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:48.327959    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl_odh-ai-gateway-infra(6b0240d5-ad57-4a49-8a88-4ee279c4d872)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872"
Jul 19 08:37:49.334911 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:49.334887    2559 scope.go:117] "RemoveContainer" containerID="2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67"
Jul 19 08:37:49.335248 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:49.335100    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl_odh-ai-gateway-infra(6b0240d5-ad57-4a49-8a88-4ee279c4d872)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872"
Jul 19 08:37:50.454908 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:50.454874    2559 scope.go:117] "RemoveContainer" containerID="4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1"
Jul 19 08:37:51.348377 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:51.348345    2559 generic.go:356] "Generic (PLEG): container finished" podID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerID="4b8373b77abf6e9157181834716c3acb66d143f538906ff3014b96d8bc9c624f" exitCode=6
Jul 19 08:37:51.348526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:51.348414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" event={"ID":"a84337bf-ff79-4ea3-913d-ee86c9afa408","Type":"ContainerDied","Data":"4b8373b77abf6e9157181834716c3acb66d143f538906ff3014b96d8bc9c624f"}
Jul 19 08:37:51.348526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:51.348475    2559 scope.go:117] "RemoveContainer" containerID="4a2e9c824357a746556e658d1bba47c3537a9fdccaddf0f4bce79ff4ec7f2bd1"
Jul 19 08:37:51.348800 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:51.348783    2559 scope.go:117] "RemoveContainer" containerID="4b8373b77abf6e9157181834716c3acb66d143f538906ff3014b96d8bc9c624f"
Jul 19 08:37:51.349063 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:37:51.349040    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv_odh-ai-gateway-infra(a84337bf-ff79-4ea3-913d-ee86c9afa408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408"
Jul 19 08:37:52.375126 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:52.375096    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:52.515559 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:52.515530    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:52.674077 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:52.673983    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca\") pod \"a84337bf-ff79-4ea3-913d-ee86c9afa408\" (UID: \"a84337bf-ff79-4ea3-913d-ee86c9afa408\") "
Jul 19 08:37:52.674388 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:52.674349    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a84337bf-ff79-4ea3-913d-ee86c9afa408" (UID: "a84337bf-ff79-4ea3-913d-ee86c9afa408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:37:52.775307 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:52.775278    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84337bf-ff79-4ea3-913d-ee86c9afa408-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:37:53.361328 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:53.361305    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"
Jul 19 08:37:53.361524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:53.361341    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv" event={"ID":"a84337bf-ff79-4ea3-913d-ee86c9afa408","Type":"ContainerDied","Data":"33ce750c2273832f03a50dfecff04fb59d044a0c7c42c07d500acbcaf76eeb1b"}
Jul 19 08:37:53.361524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:53.361396    2559 scope.go:117] "RemoveContainer" containerID="4b8373b77abf6e9157181834716c3acb66d143f538906ff3014b96d8bc9c624f"
Jul 19 08:37:53.382637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:53.382614    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:53.387631 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:53.387607    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-c2hjv"]
Jul 19 08:37:54.464878 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:37:54.463410    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" path="/var/lib/kubelet/pods/a84337bf-ff79-4ea3-913d-ee86c9afa408/volumes"
Jul 19 08:38:00.455301 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:00.455270    2559 scope.go:117] "RemoveContainer" containerID="2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67"
Jul 19 08:38:01.411378 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:01.411273    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerID="0188c6d3a8dd4a1e2b833bced0f33ba09dbdc9a6d07d0dfd80cf7d5f5a7e18b5" exitCode=6
Jul 19 08:38:01.411537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:01.411483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerDied","Data":"0188c6d3a8dd4a1e2b833bced0f33ba09dbdc9a6d07d0dfd80cf7d5f5a7e18b5"}
Jul 19 08:38:01.411611 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:01.411599    2559 scope.go:117] "RemoveContainer" containerID="2d10aa1e6c2ac1a404472da994977c08f0d128f6f090b4b7ae583449182ada67"
Jul 19 08:38:01.411943 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:01.411925    2559 scope.go:117] "RemoveContainer" containerID="0188c6d3a8dd4a1e2b833bced0f33ba09dbdc9a6d07d0dfd80cf7d5f5a7e18b5"
Jul 19 08:38:01.412148 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:01.412130    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl_odh-ai-gateway-infra(6b0240d5-ad57-4a49-8a88-4ee279c4d872)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872"
Jul 19 08:38:02.444680 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:02.444637    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:38:02.583340 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:02.583310    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:38:02.660486 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:02.660450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca\") pod \"6b0240d5-ad57-4a49-8a88-4ee279c4d872\" (UID: \"6b0240d5-ad57-4a49-8a88-4ee279c4d872\") "
Jul 19 08:38:02.660870 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:02.660849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b0240d5-ad57-4a49-8a88-4ee279c4d872" (UID: "6b0240d5-ad57-4a49-8a88-4ee279c4d872"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:38:02.761400 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:02.761324    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b0240d5-ad57-4a49-8a88-4ee279c4d872-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:38:03.424919 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:03.424888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl" event={"ID":"6b0240d5-ad57-4a49-8a88-4ee279c4d872","Type":"ContainerDied","Data":"5b98ad551115f2ad8a0c10cdebf2981c036f4cf9e779eafc612cf7863da1a1ac"}
Jul 19 08:38:03.424919 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:03.424917    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"
Jul 19 08:38:03.425177 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:03.424959    2559 scope.go:117] "RemoveContainer" containerID="0188c6d3a8dd4a1e2b833bced0f33ba09dbdc9a6d07d0dfd80cf7d5f5a7e18b5"
Jul 19 08:38:03.448250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:03.448199    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:38:03.454292 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:03.454268    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-bfhjl"]
Jul 19 08:38:04.460166 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:04.460135    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" path="/var/lib/kubelet/pods/6b0240d5-ad57-4a49-8a88-4ee279c4d872/volumes"
Jul 19 08:38:07.890870 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.890793    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891162    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891174    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891191    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891197    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891207    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891212    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891220    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891225    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891238    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891244    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf94e94-7090-4a26-8a16-5f15a60587e9" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891303    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891311    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891324    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891334    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:07.891488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.891344    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:07.895764 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.895748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:07.898254 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.898229    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:38:07.898623 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.898228    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:38:07.900977 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:07.900954    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:08.006070 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.006028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9\" (UID: \"0f1aa720-f08f-41b7-9d99-4ba9711e194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:08.107416 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.107373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9\" (UID: \"0f1aa720-f08f-41b7-9d99-4ba9711e194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:08.108033 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.108013    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9\" (UID: \"0f1aa720-f08f-41b7-9d99-4ba9711e194b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:08.207242 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.207168    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:08.340780 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.340753    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:08.342437 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:38:08.342413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1aa720_f08f_41b7_9d99_4ba9711e194b.slice/crio-8e306a280cb3a42aa4af354f9bccf15c74fda7a0e8994d28e0d40dd949c38ffb WatchSource:0}: Error finding container 8e306a280cb3a42aa4af354f9bccf15c74fda7a0e8994d28e0d40dd949c38ffb: Status 404 returned error can't find the container with id 8e306a280cb3a42aa4af354f9bccf15c74fda7a0e8994d28e0d40dd949c38ffb
Jul 19 08:38:08.462026 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:08.461948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" event={"ID":"0f1aa720-f08f-41b7-9d99-4ba9711e194b","Type":"ContainerStarted","Data":"8e306a280cb3a42aa4af354f9bccf15c74fda7a0e8994d28e0d40dd949c38ffb"}
Jul 19 08:38:09.465674 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:09.465640    2559 generic.go:356] "Generic (PLEG): container finished" podID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerID="3e2369a420f1280841af4a6c319412c0a66925da2f9a552626af2157382a0684" exitCode=6
Jul 19 08:38:09.466090 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:09.465737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" event={"ID":"0f1aa720-f08f-41b7-9d99-4ba9711e194b","Type":"ContainerDied","Data":"3e2369a420f1280841af4a6c319412c0a66925da2f9a552626af2157382a0684"}
Jul 19 08:38:09.466090 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:09.465965    2559 scope.go:117] "RemoveContainer" containerID="3e2369a420f1280841af4a6c319412c0a66925da2f9a552626af2157382a0684"
Jul 19 08:38:10.475795 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.475688    2559 generic.go:356] "Generic (PLEG): container finished" podID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerID="0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5" exitCode=6
Jul 19 08:38:10.475795 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.475746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" event={"ID":"0f1aa720-f08f-41b7-9d99-4ba9711e194b","Type":"ContainerDied","Data":"0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5"}
Jul 19 08:38:10.476270 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.475819    2559 scope.go:117] "RemoveContainer" containerID="3e2369a420f1280841af4a6c319412c0a66925da2f9a552626af2157382a0684"
Jul 19 08:38:10.476270 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.476040    2559 scope.go:117] "RemoveContainer" containerID="0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5"
Jul 19 08:38:10.476270 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:10.476256    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9_odh-ai-gateway-infra(0f1aa720-f08f-41b7-9d99-4ba9711e194b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b"
Jul 19 08:38:10.652008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.651963    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:10.652596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.652565    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:10.652596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.652589    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84337bf-ff79-4ea3-913d-ee86c9afa408" containerName="revoke-keys"
Jul 19 08:38:10.652814 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.652646    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:10.652814 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.652656    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:10.652916 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.652850    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b0240d5-ad57-4a49-8a88-4ee279c4d872" containerName="revoke-keys"
Jul 19 08:38:10.658235 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.658205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:10.660064 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.660037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:10.732015 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.731920    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g\" (UID: \"6872009b-c687-4fe4-811b-67cd0355ca9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:10.832838 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.832807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g\" (UID: \"6872009b-c687-4fe4-811b-67cd0355ca9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:10.833467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.833448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g\" (UID: \"6872009b-c687-4fe4-811b-67cd0355ca9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:10.968919 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:10.968889    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:11.104915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:11.104889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:11.106208 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:38:11.106180    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6872009b_c687_4fe4_811b_67cd0355ca9f.slice/crio-dba080acfd9f6fad519de5de083177fe3c29297fc22b82645efadb6e12872256 WatchSource:0}: Error finding container dba080acfd9f6fad519de5de083177fe3c29297fc22b82645efadb6e12872256: Status 404 returned error can't find the container with id dba080acfd9f6fad519de5de083177fe3c29297fc22b82645efadb6e12872256
Jul 19 08:38:11.485058 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:11.485028    2559 scope.go:117] "RemoveContainer" containerID="0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5"
Jul 19 08:38:11.485487 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:11.485290    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9_odh-ai-gateway-infra(0f1aa720-f08f-41b7-9d99-4ba9711e194b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b"
Jul 19 08:38:11.486241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:11.486210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerStarted","Data":"f0dbfbc42f62c7b2ef1eeafd126d576446c11e9d3a796b3bc298ee54dde69c19"}
Jul 19 08:38:11.486464 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:11.486259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerStarted","Data":"dba080acfd9f6fad519de5de083177fe3c29297fc22b82645efadb6e12872256"}
Jul 19 08:38:11.511382 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:11.511341    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" podStartSLOduration=1.51132672 podStartE2EDuration="1.51132672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:38:11.510644108 +0000 UTC m=+3328.020242009" watchObservedRunningTime="2026-07-19 08:38:11.51132672 +0000 UTC m=+3328.020924609"
Jul 19 08:38:12.492629 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:12.492594    2559 generic.go:356] "Generic (PLEG): container finished" podID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerID="f0dbfbc42f62c7b2ef1eeafd126d576446c11e9d3a796b3bc298ee54dde69c19" exitCode=6
Jul 19 08:38:12.493028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:12.492688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerDied","Data":"f0dbfbc42f62c7b2ef1eeafd126d576446c11e9d3a796b3bc298ee54dde69c19"}
Jul 19 08:38:12.493028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:12.492874    2559 scope.go:117] "RemoveContainer" containerID="f0dbfbc42f62c7b2ef1eeafd126d576446c11e9d3a796b3bc298ee54dde69c19"
Jul 19 08:38:13.499488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:13.499396    2559 generic.go:356] "Generic (PLEG): container finished" podID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerID="5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d" exitCode=6
Jul 19 08:38:13.499488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:13.499466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerDied","Data":"5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d"}
Jul 19 08:38:13.500012 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:13.499517    2559 scope.go:117] "RemoveContainer" containerID="f0dbfbc42f62c7b2ef1eeafd126d576446c11e9d3a796b3bc298ee54dde69c19"
Jul 19 08:38:13.500012 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:13.499783    2559 scope.go:117] "RemoveContainer" containerID="5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d"
Jul 19 08:38:13.500012 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:13.499960    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g_odh-ai-gateway-infra(6872009b-c687-4fe4-811b-67cd0355ca9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f"
Jul 19 08:38:14.506180 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:14.506154    2559 scope.go:117] "RemoveContainer" containerID="5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d"
Jul 19 08:38:14.506662 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:14.506369    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g_odh-ai-gateway-infra(6872009b-c687-4fe4-811b-67cd0355ca9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f"
Jul 19 08:38:23.898431 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:23.898398    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:23.903390 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:23.903372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:23.908811 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:23.908790    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:24.057969 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.057930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs\" (UID: \"02a90322-655b-44c8-85f0-05a7254be815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:24.158668 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.158592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs\" (UID: \"02a90322-655b-44c8-85f0-05a7254be815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:24.159217 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.159196    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs\" (UID: \"02a90322-655b-44c8-85f0-05a7254be815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:24.216451 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.216422    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:24.360823 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.360795    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:24.562429 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:24.562398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" event={"ID":"02a90322-655b-44c8-85f0-05a7254be815","Type":"ContainerStarted","Data":"511a66632947d005b273d9c975367bfd6aaef23ecc1e1514f5eb51a5a6ee5633"}
Jul 19 08:38:25.455272 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:25.455176    2559 scope.go:117] "RemoveContainer" containerID="5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d"
Jul 19 08:38:25.455762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:25.455350    2559 scope.go:117] "RemoveContainer" containerID="0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5"
Jul 19 08:38:25.569334 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:25.569300    2559 generic.go:356] "Generic (PLEG): container finished" podID="02a90322-655b-44c8-85f0-05a7254be815" containerID="9bba62413fdd623e5bfc9443dc6f9eeafc3be97c8900f8c62499d91fb4046763" exitCode=6
Jul 19 08:38:25.569484 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:25.569348    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" event={"ID":"02a90322-655b-44c8-85f0-05a7254be815","Type":"ContainerDied","Data":"9bba62413fdd623e5bfc9443dc6f9eeafc3be97c8900f8c62499d91fb4046763"}
Jul 19 08:38:25.569616 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:25.569603    2559 scope.go:117] "RemoveContainer" containerID="9bba62413fdd623e5bfc9443dc6f9eeafc3be97c8900f8c62499d91fb4046763"
Jul 19 08:38:26.576496 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.576456    2559 generic.go:356] "Generic (PLEG): container finished" podID="02a90322-655b-44c8-85f0-05a7254be815" containerID="ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234" exitCode=6
Jul 19 08:38:26.576960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.576518    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" event={"ID":"02a90322-655b-44c8-85f0-05a7254be815","Type":"ContainerDied","Data":"ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234"}
Jul 19 08:38:26.576960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.576577    2559 scope.go:117] "RemoveContainer" containerID="9bba62413fdd623e5bfc9443dc6f9eeafc3be97c8900f8c62499d91fb4046763"
Jul 19 08:38:26.576960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.576831    2559 scope.go:117] "RemoveContainer" containerID="ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234"
Jul 19 08:38:26.577147 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:26.577065    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs_odh-ai-gateway-infra(02a90322-655b-44c8-85f0-05a7254be815)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" podUID="02a90322-655b-44c8-85f0-05a7254be815"
Jul 19 08:38:26.578476 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.578456    2559 generic.go:356] "Generic (PLEG): container finished" podID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerID="deea2fc05cd67909c7f3595188bfc098f5d20b8a816c5f480438c0110dccd97b" exitCode=6
Jul 19 08:38:26.578559 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.578528    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerDied","Data":"deea2fc05cd67909c7f3595188bfc098f5d20b8a816c5f480438c0110dccd97b"}
Jul 19 08:38:26.578872 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.578850    2559 scope.go:117] "RemoveContainer" containerID="deea2fc05cd67909c7f3595188bfc098f5d20b8a816c5f480438c0110dccd97b"
Jul 19 08:38:26.579065 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:26.579042    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g_odh-ai-gateway-infra(6872009b-c687-4fe4-811b-67cd0355ca9f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f"
Jul 19 08:38:26.580392 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.580374    2559 generic.go:356] "Generic (PLEG): container finished" podID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerID="8dd9d2828601fda0f38f14b1239312140ad2953939a94a0e90ad270e87815b13" exitCode=6
Jul 19 08:38:26.580472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.580401    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" event={"ID":"0f1aa720-f08f-41b7-9d99-4ba9711e194b","Type":"ContainerDied","Data":"8dd9d2828601fda0f38f14b1239312140ad2953939a94a0e90ad270e87815b13"}
Jul 19 08:38:26.580626 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.580610    2559 scope.go:117] "RemoveContainer" containerID="8dd9d2828601fda0f38f14b1239312140ad2953939a94a0e90ad270e87815b13"
Jul 19 08:38:26.580795 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:26.580778    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9_odh-ai-gateway-infra(0f1aa720-f08f-41b7-9d99-4ba9711e194b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b"
Jul 19 08:38:26.590864 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.590839    2559 scope.go:117] "RemoveContainer" containerID="5e1263526c1a6455845334248ca36c6c503119bacc0f3f5071d5ac3f097c3b2d"
Jul 19 08:38:26.603116 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:26.603097    2559 scope.go:117] "RemoveContainer" containerID="0fc23c136674e49f7df4e19b55df51d820dddfcd11a1d5dca8d9a25307fed7e5"
Jul 19 08:38:27.588065 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.588031    2559 scope.go:117] "RemoveContainer" containerID="ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234"
Jul 19 08:38:27.588517 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:27.588271    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs_odh-ai-gateway-infra(02a90322-655b-44c8-85f0-05a7254be815)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" podUID="02a90322-655b-44c8-85f0-05a7254be815"
Jul 19 08:38:27.613128 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.613094    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:27.627800 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.627769    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:27.786408 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.786383    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:27.788331 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.788315    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:27.892840 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.892754    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca\") pod \"0f1aa720-f08f-41b7-9d99-4ba9711e194b\" (UID: \"0f1aa720-f08f-41b7-9d99-4ba9711e194b\") "
Jul 19 08:38:27.892840 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.892793    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca\") pod \"6872009b-c687-4fe4-811b-67cd0355ca9f\" (UID: \"6872009b-c687-4fe4-811b-67cd0355ca9f\") "
Jul 19 08:38:27.893133 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.893111    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f1aa720-f08f-41b7-9d99-4ba9711e194b" (UID: "0f1aa720-f08f-41b7-9d99-4ba9711e194b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:38:27.893220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.893199    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6872009b-c687-4fe4-811b-67cd0355ca9f" (UID: "6872009b-c687-4fe4-811b-67cd0355ca9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:38:27.993771 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.993692    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f1aa720-f08f-41b7-9d99-4ba9711e194b-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:38:27.993771 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:27.993765    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6872009b-c687-4fe4-811b-67cd0355ca9f-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:38:28.598055 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.598022    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"
Jul 19 08:38:28.598513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.598074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g" event={"ID":"6872009b-c687-4fe4-811b-67cd0355ca9f","Type":"ContainerDied","Data":"dba080acfd9f6fad519de5de083177fe3c29297fc22b82645efadb6e12872256"}
Jul 19 08:38:28.598513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.598133    2559 scope.go:117] "RemoveContainer" containerID="deea2fc05cd67909c7f3595188bfc098f5d20b8a816c5f480438c0110dccd97b"
Jul 19 08:38:28.599695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.599668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9" event={"ID":"0f1aa720-f08f-41b7-9d99-4ba9711e194b","Type":"ContainerDied","Data":"8e306a280cb3a42aa4af354f9bccf15c74fda7a0e8994d28e0d40dd949c38ffb"}
Jul 19 08:38:28.599695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.599686    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"
Jul 19 08:38:28.612037 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.611895    2559 scope.go:117] "RemoveContainer" containerID="8dd9d2828601fda0f38f14b1239312140ad2953939a94a0e90ad270e87815b13"
Jul 19 08:38:28.618298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.618270    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:28.628323 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.628301    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-tl59g"]
Jul 19 08:38:28.640012 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.639993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:28.646033 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:28.646011    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-tzcg9"]
Jul 19 08:38:30.460717 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:30.460678    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" path="/var/lib/kubelet/pods/0f1aa720-f08f-41b7-9d99-4ba9711e194b/volumes"
Jul 19 08:38:30.461079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:30.461057    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" path="/var/lib/kubelet/pods/6872009b-c687-4fe4-811b-67cd0355ca9f/volumes"
Jul 19 08:38:35.591427 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591391    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591754    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591765    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591774    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591780    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591792    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591797    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591806    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591811    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591826    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.591825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591832    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.592137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591884    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.592137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591891    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.592137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591899    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:38:35.592137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591904    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.592137 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.591911    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:38:35.596261 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.596245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:35.601803 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.601775    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:35.763943 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.763908    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw\" (UID: \"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:35.864757 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.864656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw\" (UID: \"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:35.865337 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.865314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw\" (UID: \"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:35.910201 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:35.910173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:36.040364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:36.040334    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:36.043781 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:38:36.043754    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a2c7b5_fec6_4060_84fb_47d6b0b3a4f0.slice/crio-47341827650dd2a11a54ae5477f73f1dd62c7f45ed49d463cdbc571e6fd82d88 WatchSource:0}: Error finding container 47341827650dd2a11a54ae5477f73f1dd62c7f45ed49d463cdbc571e6fd82d88: Status 404 returned error can't find the container with id 47341827650dd2a11a54ae5477f73f1dd62c7f45ed49d463cdbc571e6fd82d88
Jul 19 08:38:36.648295 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:36.648257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerStarted","Data":"2b169fbcfe9180bbbd78fccbbb3d254007fa6b17f55a4961f9b0efd9227e2384"}
Jul 19 08:38:36.648782 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:36.648316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerStarted","Data":"47341827650dd2a11a54ae5477f73f1dd62c7f45ed49d463cdbc571e6fd82d88"}
Jul 19 08:38:36.664118 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:36.664071    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" podStartSLOduration=1.664056797 podStartE2EDuration="1.664056797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:38:36.661881177 +0000 UTC m=+3353.171479076" watchObservedRunningTime="2026-07-19 08:38:36.664056797 +0000 UTC m=+3353.173654696"
Jul 19 08:38:37.658161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:37.658130    2559 generic.go:356] "Generic (PLEG): container finished" podID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerID="2b169fbcfe9180bbbd78fccbbb3d254007fa6b17f55a4961f9b0efd9227e2384" exitCode=6
Jul 19 08:38:37.658572 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:37.658222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerDied","Data":"2b169fbcfe9180bbbd78fccbbb3d254007fa6b17f55a4961f9b0efd9227e2384"}
Jul 19 08:38:37.658572 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:37.658387    2559 scope.go:117] "RemoveContainer" containerID="2b169fbcfe9180bbbd78fccbbb3d254007fa6b17f55a4961f9b0efd9227e2384"
Jul 19 08:38:38.665916 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:38.665886    2559 generic.go:356] "Generic (PLEG): container finished" podID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerID="747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022" exitCode=6
Jul 19 08:38:38.666420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:38.665967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerDied","Data":"747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022"}
Jul 19 08:38:38.666420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:38.666030    2559 scope.go:117] "RemoveContainer" containerID="2b169fbcfe9180bbbd78fccbbb3d254007fa6b17f55a4961f9b0efd9227e2384"
Jul 19 08:38:38.666420 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:38.666269    2559 scope.go:117] "RemoveContainer" containerID="747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022"
Jul 19 08:38:38.666592 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:38.666538    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw_odh-ai-gateway-infra(39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0"
Jul 19 08:38:39.455028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:39.454995    2559 scope.go:117] "RemoveContainer" containerID="ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234"
Jul 19 08:38:39.673800 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:39.673774    2559 scope.go:117] "RemoveContainer" containerID="747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022"
Jul 19 08:38:39.674128 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:39.674043    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw_odh-ai-gateway-infra(39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0"
Jul 19 08:38:40.683472 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:40.683438    2559 generic.go:356] "Generic (PLEG): container finished" podID="02a90322-655b-44c8-85f0-05a7254be815" containerID="205dc073ed9140f930f3a9e6371a74b878fabc29396dedc40cef2ae92938ad8d" exitCode=6
Jul 19 08:38:40.683910 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:40.683514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" event={"ID":"02a90322-655b-44c8-85f0-05a7254be815","Type":"ContainerDied","Data":"205dc073ed9140f930f3a9e6371a74b878fabc29396dedc40cef2ae92938ad8d"}
Jul 19 08:38:40.683910 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:40.683578    2559 scope.go:117] "RemoveContainer" containerID="ba1ff87a4e09dfb44dabb26c5ced56946df0cf5542cb815bec09cae6a13e9234"
Jul 19 08:38:40.684035 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:40.683977    2559 scope.go:117] "RemoveContainer" containerID="205dc073ed9140f930f3a9e6371a74b878fabc29396dedc40cef2ae92938ad8d"
Jul 19 08:38:40.684212 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:40.684191    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs_odh-ai-gateway-infra(02a90322-655b-44c8-85f0-05a7254be815)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" podUID="02a90322-655b-44c8-85f0-05a7254be815"
Jul 19 08:38:41.713524 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:41.713494    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:41.840988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:41.840964    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:41.920673 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:41.920640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca\") pod \"02a90322-655b-44c8-85f0-05a7254be815\" (UID: \"02a90322-655b-44c8-85f0-05a7254be815\") "
Jul 19 08:38:41.921028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:41.921004    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02a90322-655b-44c8-85f0-05a7254be815" (UID: "02a90322-655b-44c8-85f0-05a7254be815"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:38:42.021296 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.021215    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02a90322-655b-44c8-85f0-05a7254be815-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:38:42.697605 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.697578    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"
Jul 19 08:38:42.697795 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.697580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs" event={"ID":"02a90322-655b-44c8-85f0-05a7254be815","Type":"ContainerDied","Data":"511a66632947d005b273d9c975367bfd6aaef23ecc1e1514f5eb51a5a6ee5633"}
Jul 19 08:38:42.697795 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.697761    2559 scope.go:117] "RemoveContainer" containerID="205dc073ed9140f930f3a9e6371a74b878fabc29396dedc40cef2ae92938ad8d"
Jul 19 08:38:42.714297 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.714271    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:42.717879 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:42.717858    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-6lsbs"]
Jul 19 08:38:44.463154 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:44.463124    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a90322-655b-44c8-85f0-05a7254be815" path="/var/lib/kubelet/pods/02a90322-655b-44c8-85f0-05a7254be815/volumes"
Jul 19 08:38:54.460604 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:54.460574    2559 scope.go:117] "RemoveContainer" containerID="747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022"
Jul 19 08:38:55.771404 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:55.771370    2559 generic.go:356] "Generic (PLEG): container finished" podID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerID="35724964aadc9ae9a978f5d958445ec63b5dce99bf540c11f6442e345ec2215a" exitCode=6
Jul 19 08:38:55.771939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:55.771444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerDied","Data":"35724964aadc9ae9a978f5d958445ec63b5dce99bf540c11f6442e345ec2215a"}
Jul 19 08:38:55.771939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:55.771524    2559 scope.go:117] "RemoveContainer" containerID="747066bb56e3dd5a53931d3bf2952bea89b60a4057731d3fd43f7226ca1c9022"
Jul 19 08:38:55.771939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:55.771843    2559 scope.go:117] "RemoveContainer" containerID="35724964aadc9ae9a978f5d958445ec63b5dce99bf540c11f6442e345ec2215a"
Jul 19 08:38:55.772068 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:38:55.772044    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw_odh-ai-gateway-infra(39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0"
Jul 19 08:38:56.798101 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:56.798072    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:56.929341 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:56.929317    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:57.052243 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.052152    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca\") pod \"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0\" (UID: \"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0\") "
Jul 19 08:38:57.052552 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.052525    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" (UID: "39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:38:57.153174 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.153136    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:38:57.784915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.784887    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"
Jul 19 08:38:57.784915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.784913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw" event={"ID":"39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0","Type":"ContainerDied","Data":"47341827650dd2a11a54ae5477f73f1dd62c7f45ed49d463cdbc571e6fd82d88"}
Jul 19 08:38:57.785174 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.784965    2559 scope.go:117] "RemoveContainer" containerID="35724964aadc9ae9a978f5d958445ec63b5dce99bf540c11f6442e345ec2215a"
Jul 19 08:38:57.807999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.807975    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:57.815896 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:57.815872    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-dpvtw"]
Jul 19 08:38:58.460659 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:38:58.460629    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" path="/var/lib/kubelet/pods/39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0/volumes"
Jul 19 08:39:00.692993 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.692958    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693301    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693312    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693332    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693338    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693344    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693349    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693358    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693363    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693373    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693379    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1aa720-f08f-41b7-9d99-4ba9711e194b" containerName="revoke-keys"
Jul 19 08:39:00.693383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693386    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693391    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693400    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693405    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693454    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693463    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693468    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693473    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6872009b-c687-4fe4-811b-67cd0355ca9f" containerName="revoke-keys"
Jul 19 08:39:00.693763 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.693480    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:00.697729 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.697692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:00.700846 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.700822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:39:00.701519 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.701266    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:39:00.703990 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.703967    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:00.785475 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.785449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9\" (UID: \"ad21c9b2-c826-415d-bcc8-2e31d59aaee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:00.886854 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.886818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9\" (UID: \"ad21c9b2-c826-415d-bcc8-2e31d59aaee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:00.887443 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:00.887425    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9\" (UID: \"ad21c9b2-c826-415d-bcc8-2e31d59aaee9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:01.009439 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:01.009359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:01.139208 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:01.139182    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:01.141922 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:39:01.141895    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad21c9b2_c826_415d_bcc8_2e31d59aaee9.slice/crio-872723a5053b82e7627d1f891420aba167dc9c4d3f3f15a1cafe09b6f5ce7933 WatchSource:0}: Error finding container 872723a5053b82e7627d1f891420aba167dc9c4d3f3f15a1cafe09b6f5ce7933: Status 404 returned error can't find the container with id 872723a5053b82e7627d1f891420aba167dc9c4d3f3f15a1cafe09b6f5ce7933
Jul 19 08:39:01.808297 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:01.808214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerStarted","Data":"b00d283337d1a964351e67f1226deae02640dadbcf1741fb99c80f04471c3dd8"}
Jul 19 08:39:01.808297 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:01.808295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerStarted","Data":"872723a5053b82e7627d1f891420aba167dc9c4d3f3f15a1cafe09b6f5ce7933"}
Jul 19 08:39:01.826349 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:01.826298    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" podStartSLOduration=1.826283243 podStartE2EDuration="1.826283243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:39:01.824597695 +0000 UTC m=+3378.334195595" watchObservedRunningTime="2026-07-19 08:39:01.826283243 +0000 UTC m=+3378.335881141"
Jul 19 08:39:02.814894 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:02.814854    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerID="b00d283337d1a964351e67f1226deae02640dadbcf1741fb99c80f04471c3dd8" exitCode=6
Jul 19 08:39:02.815391 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:02.814919    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerDied","Data":"b00d283337d1a964351e67f1226deae02640dadbcf1741fb99c80f04471c3dd8"}
Jul 19 08:39:02.815391 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:02.815172    2559 scope.go:117] "RemoveContainer" containerID="b00d283337d1a964351e67f1226deae02640dadbcf1741fb99c80f04471c3dd8"
Jul 19 08:39:03.822923 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:03.822887    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerID="cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b" exitCode=6
Jul 19 08:39:03.823339 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:03.822967    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerDied","Data":"cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b"}
Jul 19 08:39:03.823339 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:03.823032    2559 scope.go:117] "RemoveContainer" containerID="b00d283337d1a964351e67f1226deae02640dadbcf1741fb99c80f04471c3dd8"
Jul 19 08:39:03.823339 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:03.823221    2559 scope.go:117] "RemoveContainer" containerID="cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b"
Jul 19 08:39:03.823508 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:03.823490    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9_odh-ai-gateway-infra(ad21c9b2-c826-415d-bcc8-2e31d59aaee9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9"
Jul 19 08:39:04.830762 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:04.830737    2559 scope.go:117] "RemoveContainer" containerID="cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b"
Jul 19 08:39:04.831131 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:04.830929    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9_odh-ai-gateway-infra(ad21c9b2-c826-415d-bcc8-2e31d59aaee9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9"
Jul 19 08:39:07.936721 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:07.936664    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:07.937148 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:07.937133    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a90322-655b-44c8-85f0-05a7254be815" containerName="revoke-keys"
Jul 19 08:39:07.937195 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:07.937150    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a2c7b5-fec6-4060-84fb-47d6b0b3a4f0" containerName="revoke-keys"
Jul 19 08:39:07.941396 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:07.941379    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:07.949693 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:07.949193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:08.051008 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.050969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-cljmf\" (UID: \"fdcdfedd-844e-417e-8502-f0153d923c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:08.152206 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.152172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-cljmf\" (UID: \"fdcdfedd-844e-417e-8502-f0153d923c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:08.152880 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.152862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-cljmf\" (UID: \"fdcdfedd-844e-417e-8502-f0153d923c78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:08.254869 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.254783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:08.384006 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.383979    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:08.386067 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:39:08.386040    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcdfedd_844e_417e_8502_f0153d923c78.slice/crio-b812c0e8bed30bd4a5ab1d22bfd0813e94e0ef54c06ddd9df65f0ef6db35c37f WatchSource:0}: Error finding container b812c0e8bed30bd4a5ab1d22bfd0813e94e0ef54c06ddd9df65f0ef6db35c37f: Status 404 returned error can't find the container with id b812c0e8bed30bd4a5ab1d22bfd0813e94e0ef54c06ddd9df65f0ef6db35c37f
Jul 19 08:39:08.853887 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.853799    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerStarted","Data":"ce2be14f0af7a3664c30e1fd83b84e9a1fe6ff612a03537e8ad016f107660f81"}
Jul 19 08:39:08.853887 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.853866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerStarted","Data":"b812c0e8bed30bd4a5ab1d22bfd0813e94e0ef54c06ddd9df65f0ef6db35c37f"}
Jul 19 08:39:08.868083 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:08.867965    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" podStartSLOduration=1.867945863 podStartE2EDuration="1.867945863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:39:08.867185723 +0000 UTC m=+3385.376783622" watchObservedRunningTime="2026-07-19 08:39:08.867945863 +0000 UTC m=+3385.377543763"
Jul 19 08:39:09.861004 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:09.860972    2559 generic.go:356] "Generic (PLEG): container finished" podID="fdcdfedd-844e-417e-8502-f0153d923c78" containerID="ce2be14f0af7a3664c30e1fd83b84e9a1fe6ff612a03537e8ad016f107660f81" exitCode=6
Jul 19 08:39:09.861419 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:09.861062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerDied","Data":"ce2be14f0af7a3664c30e1fd83b84e9a1fe6ff612a03537e8ad016f107660f81"}
Jul 19 08:39:09.861419 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:09.861226    2559 scope.go:117] "RemoveContainer" containerID="ce2be14f0af7a3664c30e1fd83b84e9a1fe6ff612a03537e8ad016f107660f81"
Jul 19 08:39:10.868778 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:10.868749    2559 generic.go:356] "Generic (PLEG): container finished" podID="fdcdfedd-844e-417e-8502-f0153d923c78" containerID="339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd" exitCode=6
Jul 19 08:39:10.869202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:10.868820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerDied","Data":"339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd"}
Jul 19 08:39:10.869202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:10.868866    2559 scope.go:117] "RemoveContainer" containerID="ce2be14f0af7a3664c30e1fd83b84e9a1fe6ff612a03537e8ad016f107660f81"
Jul 19 08:39:10.869202 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:10.869089    2559 scope.go:117] "RemoveContainer" containerID="339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd"
Jul 19 08:39:10.869334 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:10.869323    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-cljmf_odh-ai-gateway-infra(fdcdfedd-844e-417e-8502-f0153d923c78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" podUID="fdcdfedd-844e-417e-8502-f0153d923c78"
Jul 19 08:39:11.875925 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:11.875895    2559 scope.go:117] "RemoveContainer" containerID="339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd"
Jul 19 08:39:11.876312 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:11.876126    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-cljmf_odh-ai-gateway-infra(fdcdfedd-844e-417e-8502-f0153d923c78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" podUID="fdcdfedd-844e-417e-8502-f0153d923c78"
Jul 19 08:39:13.941660 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:13.941628    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:13.946326 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:13.946307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:13.951425 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:13.951398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:14.106871 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.106838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5\" (UID: \"3b60c117-62de-4ad6-aa5f-1e8f2c581217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:14.208205 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.208121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5\" (UID: \"3b60c117-62de-4ad6-aa5f-1e8f2c581217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:14.208695 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.208677    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5\" (UID: \"3b60c117-62de-4ad6-aa5f-1e8f2c581217\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:14.258117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.258089    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:14.388100 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.388076    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:14.389676 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:39:14.389650    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b60c117_62de_4ad6_aa5f_1e8f2c581217.slice/crio-b403a8f26f336c849c05aedb9762ef644360b0d0143e889590fb2192ea6f368e WatchSource:0}: Error finding container b403a8f26f336c849c05aedb9762ef644360b0d0143e889590fb2192ea6f368e: Status 404 returned error can't find the container with id b403a8f26f336c849c05aedb9762ef644360b0d0143e889590fb2192ea6f368e
Jul 19 08:39:14.893399 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.893321    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerStarted","Data":"5c3daf2586102ab404de8688e0ad2ba477cb800a59a60b11ad8d3888f987484c"}
Jul 19 08:39:14.893399 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.893385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerStarted","Data":"b403a8f26f336c849c05aedb9762ef644360b0d0143e889590fb2192ea6f368e"}
Jul 19 08:39:14.908945 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:14.908899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" podStartSLOduration=1.908884503 podStartE2EDuration="1.908884503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:39:14.907276596 +0000 UTC m=+3391.416874496" watchObservedRunningTime="2026-07-19 08:39:14.908884503 +0000 UTC m=+3391.418482401"
Jul 19 08:39:15.900743 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:15.900679    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerID="5c3daf2586102ab404de8688e0ad2ba477cb800a59a60b11ad8d3888f987484c" exitCode=6
Jul 19 08:39:15.901117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:15.900750    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerDied","Data":"5c3daf2586102ab404de8688e0ad2ba477cb800a59a60b11ad8d3888f987484c"}
Jul 19 08:39:15.901117 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:15.901058    2559 scope.go:117] "RemoveContainer" containerID="5c3daf2586102ab404de8688e0ad2ba477cb800a59a60b11ad8d3888f987484c"
Jul 19 08:39:16.908484 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:16.908449    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerID="0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77" exitCode=6
Jul 19 08:39:16.908945 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:16.908530    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerDied","Data":"0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77"}
Jul 19 08:39:16.908945 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:16.908596    2559 scope.go:117] "RemoveContainer" containerID="5c3daf2586102ab404de8688e0ad2ba477cb800a59a60b11ad8d3888f987484c"
Jul 19 08:39:16.908945 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:16.908836    2559 scope.go:117] "RemoveContainer" containerID="0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77"
Jul 19 08:39:16.909113 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:16.909064    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5_odh-ai-gateway-infra(3b60c117-62de-4ad6-aa5f-1e8f2c581217)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217"
Jul 19 08:39:17.455543 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:17.455508    2559 scope.go:117] "RemoveContainer" containerID="cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b"
Jul 19 08:39:17.915951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:17.915911    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerStarted","Data":"72f28ee32b63f3b79ea7dbdec515934724cda43d29e73167a48a2613e31d3996"}
Jul 19 08:39:17.917646 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:17.917630    2559 scope.go:117] "RemoveContainer" containerID="0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77"
Jul 19 08:39:17.917841 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:17.917823    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5_odh-ai-gateway-infra(3b60c117-62de-4ad6-aa5f-1e8f2c581217)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217"
Jul 19 08:39:18.925364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:18.925334    2559 generic.go:356] "Generic (PLEG): container finished" podID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerID="72f28ee32b63f3b79ea7dbdec515934724cda43d29e73167a48a2613e31d3996" exitCode=6
Jul 19 08:39:18.925811 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:18.925387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerDied","Data":"72f28ee32b63f3b79ea7dbdec515934724cda43d29e73167a48a2613e31d3996"}
Jul 19 08:39:18.925811 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:18.925434    2559 scope.go:117] "RemoveContainer" containerID="cbe5396db9d57a1fe6f4187307d526ff786453044cd544f5ccbff63c9f8c452b"
Jul 19 08:39:18.925811 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:18.925753    2559 scope.go:117] "RemoveContainer" containerID="72f28ee32b63f3b79ea7dbdec515934724cda43d29e73167a48a2613e31d3996"
Jul 19 08:39:18.925995 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:18.925972    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9_odh-ai-gateway-infra(ad21c9b2-c826-415d-bcc8-2e31d59aaee9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9"
Jul 19 08:39:18.940299 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:18.940278    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:20.068368 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.068340    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:20.165196 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.165164    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca\") pod \"ad21c9b2-c826-415d-bcc8-2e31d59aaee9\" (UID: \"ad21c9b2-c826-415d-bcc8-2e31d59aaee9\") "
Jul 19 08:39:20.165499 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.165477    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad21c9b2-c826-415d-bcc8-2e31d59aaee9" (UID: "ad21c9b2-c826-415d-bcc8-2e31d59aaee9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:39:20.266663 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.266582    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad21c9b2-c826-415d-bcc8-2e31d59aaee9-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:39:20.938518 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.938488    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"
Jul 19 08:39:20.938518 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.938505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9" event={"ID":"ad21c9b2-c826-415d-bcc8-2e31d59aaee9","Type":"ContainerDied","Data":"872723a5053b82e7627d1f891420aba167dc9c4d3f3f15a1cafe09b6f5ce7933"}
Jul 19 08:39:20.938798 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.938586    2559 scope.go:117] "RemoveContainer" containerID="72f28ee32b63f3b79ea7dbdec515934724cda43d29e73167a48a2613e31d3996"
Jul 19 08:39:20.955423 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.955399    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:20.959825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:20.959804    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-7ngf9"]
Jul 19 08:39:22.460917 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:22.460883    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" path="/var/lib/kubelet/pods/ad21c9b2-c826-415d-bcc8-2e31d59aaee9/volumes"
Jul 19 08:39:23.455759 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:23.455731    2559 scope.go:117] "RemoveContainer" containerID="339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd"
Jul 19 08:39:23.957429 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:23.957397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerStarted","Data":"aa76cf516bc723e0527c576723fe6b8395341e04b09ffcffbbb1d9230211f86c"}
Jul 19 08:39:24.964103 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:24.964072    2559 generic.go:356] "Generic (PLEG): container finished" podID="fdcdfedd-844e-417e-8502-f0153d923c78" containerID="aa76cf516bc723e0527c576723fe6b8395341e04b09ffcffbbb1d9230211f86c" exitCode=6
Jul 19 08:39:24.964507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:24.964147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerDied","Data":"aa76cf516bc723e0527c576723fe6b8395341e04b09ffcffbbb1d9230211f86c"}
Jul 19 08:39:24.964507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:24.964213    2559 scope.go:117] "RemoveContainer" containerID="339a3eb64fb90392ba74f5edc1e54cc1dc7d09e91f30c915baf18791f9d89dcd"
Jul 19 08:39:24.964507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:24.964504    2559 scope.go:117] "RemoveContainer" containerID="aa76cf516bc723e0527c576723fe6b8395341e04b09ffcffbbb1d9230211f86c"
Jul 19 08:39:24.964783 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:24.964760    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-cljmf_odh-ai-gateway-infra(fdcdfedd-844e-417e-8502-f0153d923c78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" podUID="fdcdfedd-844e-417e-8502-f0153d923c78"
Jul 19 08:39:24.985244 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:24.985218    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:26.110770 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.110741    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:26.220467 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.220425    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca\") pod \"fdcdfedd-844e-417e-8502-f0153d923c78\" (UID: \"fdcdfedd-844e-417e-8502-f0153d923c78\") "
Jul 19 08:39:26.220859 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.220837    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdcdfedd-844e-417e-8502-f0153d923c78" (UID: "fdcdfedd-844e-417e-8502-f0153d923c78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:39:26.321173 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.321145    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdcdfedd-844e-417e-8502-f0153d923c78-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:39:26.978272 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.978235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf" event={"ID":"fdcdfedd-844e-417e-8502-f0153d923c78","Type":"ContainerDied","Data":"b812c0e8bed30bd4a5ab1d22bfd0813e94e0ef54c06ddd9df65f0ef6db35c37f"}
Jul 19 08:39:26.978465 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.978302    2559 scope.go:117] "RemoveContainer" containerID="aa76cf516bc723e0527c576723fe6b8395341e04b09ffcffbbb1d9230211f86c"
Jul 19 08:39:26.978465 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.978251    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"
Jul 19 08:39:26.994364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:26.994334    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:27.000250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:27.000226    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-cljmf"]
Jul 19 08:39:28.461251 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:28.461219    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" path="/var/lib/kubelet/pods/fdcdfedd-844e-417e-8502-f0153d923c78/volumes"
Jul 19 08:39:31.455143 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:31.455114    2559 scope.go:117] "RemoveContainer" containerID="0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77"
Jul 19 08:39:32.007025 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:32.006989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerStarted","Data":"39ebfa39e411530fa02afef1c4cac909e345d8cbf7e31628a38b924c125d936e"}
Jul 19 08:39:33.013988 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:33.013955    2559 generic.go:356] "Generic (PLEG): container finished" podID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerID="39ebfa39e411530fa02afef1c4cac909e345d8cbf7e31628a38b924c125d936e" exitCode=6
Jul 19 08:39:33.014412 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:33.014027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerDied","Data":"39ebfa39e411530fa02afef1c4cac909e345d8cbf7e31628a38b924c125d936e"}
Jul 19 08:39:33.014412 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:33.014100    2559 scope.go:117] "RemoveContainer" containerID="0632fcc6e9f0c2418b45d142c0a15330850f185ed8cd68dacb4793d696f89b77"
Jul 19 08:39:33.014537 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:33.014437    2559 scope.go:117] "RemoveContainer" containerID="39ebfa39e411530fa02afef1c4cac909e345d8cbf7e31628a38b924c125d936e"
Jul 19 08:39:33.014771 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:33.014741    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5_odh-ai-gateway-infra(3b60c117-62de-4ad6-aa5f-1e8f2c581217)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217"
Jul 19 08:39:33.035884 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:33.033406    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:34.152307 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:34.152283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:34.193170 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:34.193140    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca\") pod \"3b60c117-62de-4ad6-aa5f-1e8f2c581217\" (UID: \"3b60c117-62de-4ad6-aa5f-1e8f2c581217\") "
Jul 19 08:39:34.193462 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:34.193442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b60c117-62de-4ad6-aa5f-1e8f2c581217" (UID: "3b60c117-62de-4ad6-aa5f-1e8f2c581217"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:39:34.294489 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:34.294415    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b60c117-62de-4ad6-aa5f-1e8f2c581217-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:39:35.027222 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.027194    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"
Jul 19 08:39:35.027427 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.027193    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5" event={"ID":"3b60c117-62de-4ad6-aa5f-1e8f2c581217","Type":"ContainerDied","Data":"b403a8f26f336c849c05aedb9762ef644360b0d0143e889590fb2192ea6f368e"}
Jul 19 08:39:35.027427 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.027341    2559 scope.go:117] "RemoveContainer" containerID="39ebfa39e411530fa02afef1c4cac909e345d8cbf7e31628a38b924c125d936e"
Jul 19 08:39:35.043205 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.043179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:35.048392 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.048372    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-v2sq5"]
Jul 19 08:39:35.640161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640083    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640425    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640435    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640445    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640452    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640460    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640466    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640473    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640478    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640484    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640490    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640504    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640509    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640515    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640523 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640520    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640580    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640588    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640594    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640600    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640605    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:35.640967 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.640612    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:35.645002 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.644983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:35.647383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.647364    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:39:35.647540 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.647373    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:39:35.652187 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.652164    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:35.704808 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.704777    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj\" (UID: \"7e567e4f-e72e-4a6d-b9c4-73a6b5644822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:35.805391 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.805357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj\" (UID: \"7e567e4f-e72e-4a6d-b9c4-73a6b5644822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:35.806009 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.805989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj\" (UID: \"7e567e4f-e72e-4a6d-b9c4-73a6b5644822\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:35.957013 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:35.956929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:36.105743 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:36.105719    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:36.107433 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:39:36.107409    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e567e4f_e72e_4a6d_b9c4_73a6b5644822.slice/crio-237930aa6f02e39369cf7a6780fc89e95f9b2b4c50978f7eb79c1a56b03a5d05 WatchSource:0}: Error finding container 237930aa6f02e39369cf7a6780fc89e95f9b2b4c50978f7eb79c1a56b03a5d05: Status 404 returned error can't find the container with id 237930aa6f02e39369cf7a6780fc89e95f9b2b4c50978f7eb79c1a56b03a5d05
Jul 19 08:39:36.461438 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:36.461408    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" path="/var/lib/kubelet/pods/3b60c117-62de-4ad6-aa5f-1e8f2c581217/volumes"
Jul 19 08:39:37.042222 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:37.042129    2559 generic.go:356] "Generic (PLEG): container finished" podID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerID="e81094014d78d76e47b7287fa93166d4f5bd73a93ea70b2d9f012e6228700ef5" exitCode=6
Jul 19 08:39:37.042602 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:37.042215    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerDied","Data":"e81094014d78d76e47b7287fa93166d4f5bd73a93ea70b2d9f012e6228700ef5"}
Jul 19 08:39:37.042602 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:37.042298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerStarted","Data":"237930aa6f02e39369cf7a6780fc89e95f9b2b4c50978f7eb79c1a56b03a5d05"}
Jul 19 08:39:37.042602 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:37.042362    2559 scope.go:117] "RemoveContainer" containerID="e81094014d78d76e47b7287fa93166d4f5bd73a93ea70b2d9f012e6228700ef5"
Jul 19 08:39:38.050463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:38.050380    2559 generic.go:356] "Generic (PLEG): container finished" podID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerID="b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5" exitCode=6
Jul 19 08:39:38.050939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:38.050476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerDied","Data":"b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5"}
Jul 19 08:39:38.050939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:38.050548    2559 scope.go:117] "RemoveContainer" containerID="e81094014d78d76e47b7287fa93166d4f5bd73a93ea70b2d9f012e6228700ef5"
Jul 19 08:39:38.050939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:38.050628    2559 scope.go:117] "RemoveContainer" containerID="b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5"
Jul 19 08:39:38.050939 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:38.050874    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj_odh-ai-gateway-infra(7e567e4f-e72e-4a6d-b9c4-73a6b5644822)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822"
Jul 19 08:39:39.057718 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:39.057684    2559 scope.go:117] "RemoveContainer" containerID="b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5"
Jul 19 08:39:39.058122 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:39.057931    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj_odh-ai-gateway-infra(7e567e4f-e72e-4a6d-b9c4-73a6b5644822)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822"
Jul 19 08:39:51.454788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:51.454759    2559 scope.go:117] "RemoveContainer" containerID="b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5"
Jul 19 08:39:52.134853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:52.134816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerStarted","Data":"59afba8ab6a0196c2b5ea19383a793c13bfb07980fb783ade63d00a7cd6e03c5"}
Jul 19 08:39:52.152283 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:52.152227    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" podStartSLOduration=17.152211311 podStartE2EDuration="17.152211311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:39:52.150869283 +0000 UTC m=+3428.660467182" watchObservedRunningTime="2026-07-19 08:39:52.152211311 +0000 UTC m=+3428.661809210"
Jul 19 08:39:53.142514 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:53.142481    2559 generic.go:356] "Generic (PLEG): container finished" podID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerID="59afba8ab6a0196c2b5ea19383a793c13bfb07980fb783ade63d00a7cd6e03c5" exitCode=6
Jul 19 08:39:53.142951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:53.142550    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerDied","Data":"59afba8ab6a0196c2b5ea19383a793c13bfb07980fb783ade63d00a7cd6e03c5"}
Jul 19 08:39:53.142951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:53.142610    2559 scope.go:117] "RemoveContainer" containerID="b68694ceb69046ef1e479207b1c0a958d59a7d8d45e94a2136ecd5552f0ec6a5"
Jul 19 08:39:53.142951 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:53.142867    2559 scope.go:117] "RemoveContainer" containerID="59afba8ab6a0196c2b5ea19383a793c13bfb07980fb783ade63d00a7cd6e03c5"
Jul 19 08:39:53.143107 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:39:53.143085    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj_odh-ai-gateway-infra(7e567e4f-e72e-4a6d-b9c4-73a6b5644822)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822"
Jul 19 08:39:53.161007 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:53.160982    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:54.283448 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:54.283422    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:54.373724 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:54.373673    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca\") pod \"7e567e4f-e72e-4a6d-b9c4-73a6b5644822\" (UID: \"7e567e4f-e72e-4a6d-b9c4-73a6b5644822\") "
Jul 19 08:39:54.374075 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:54.374054    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e567e4f-e72e-4a6d-b9c4-73a6b5644822" (UID: "7e567e4f-e72e-4a6d-b9c4-73a6b5644822"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:39:54.474359 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:54.474337    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e567e4f-e72e-4a6d-b9c4-73a6b5644822-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:39:55.157358 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:55.157332    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"
Jul 19 08:39:55.157516 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:55.157368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj" event={"ID":"7e567e4f-e72e-4a6d-b9c4-73a6b5644822","Type":"ContainerDied","Data":"237930aa6f02e39369cf7a6780fc89e95f9b2b4c50978f7eb79c1a56b03a5d05"}
Jul 19 08:39:55.157516 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:55.157430    2559 scope.go:117] "RemoveContainer" containerID="59afba8ab6a0196c2b5ea19383a793c13bfb07980fb783ade63d00a7cd6e03c5"
Jul 19 08:39:55.174501 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:55.174475    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:55.177635 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:55.177610    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-v4xpj"]
Jul 19 08:39:56.460831 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:56.460802    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" path="/var/lib/kubelet/pods/7e567e4f-e72e-4a6d-b9c4-73a6b5644822/volumes"
Jul 19 08:39:57.980611 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980573    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980957    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980968    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980981    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980986    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.980999    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981005    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981010    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981015    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981024    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981029    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981078    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981086    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981093    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b60c117-62de-4ad6-aa5f-1e8f2c581217" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981100    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad21c9b2-c826-415d-bcc8-2e31d59aaee9" containerName="revoke-keys"
Jul 19 08:39:57.981107 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.981106    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdcdfedd-844e-417e-8502-f0153d923c78" containerName="revoke-keys"
Jul 19 08:39:57.990012 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.989298    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:39:57.991982 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.991754    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:39:57.991982 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.991771    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:39:57.992796 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:57.992770    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:39:58.106789 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:58.106752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-b6flx\" (UID: \"38250c63-a420-4274-a5f2-29896f80892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:39:58.208028 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:58.207997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-b6flx\" (UID: \"38250c63-a420-4274-a5f2-29896f80892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:39:58.208599 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:58.208579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef124b28-b6flx\" (UID: \"38250c63-a420-4274-a5f2-29896f80892f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:39:58.301073 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:58.300997    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:39:58.430477 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:58.430449    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:39:58.431748 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:39:58.431722    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38250c63_a420_4274_a5f2_29896f80892f.slice/crio-bd09808cd32a6a60777e29ec34b34623ee416514f0468e8cf0a9c2326b289612 WatchSource:0}: Error finding container bd09808cd32a6a60777e29ec34b34623ee416514f0468e8cf0a9c2326b289612: Status 404 returned error can't find the container with id bd09808cd32a6a60777e29ec34b34623ee416514f0468e8cf0a9c2326b289612
Jul 19 08:39:59.180663 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:59.180635    2559 generic.go:356] "Generic (PLEG): container finished" podID="38250c63-a420-4274-a5f2-29896f80892f" containerID="c98cbd4b215785997b751cbfcbea0367e4987945d0099617c8f031b71aa62268" exitCode=6
Jul 19 08:39:59.181079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:59.180734    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" event={"ID":"38250c63-a420-4274-a5f2-29896f80892f","Type":"ContainerDied","Data":"c98cbd4b215785997b751cbfcbea0367e4987945d0099617c8f031b71aa62268"}
Jul 19 08:39:59.181079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:59.180802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" event={"ID":"38250c63-a420-4274-a5f2-29896f80892f","Type":"ContainerStarted","Data":"bd09808cd32a6a60777e29ec34b34623ee416514f0468e8cf0a9c2326b289612"}
Jul 19 08:39:59.181079 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:39:59.180872    2559 scope.go:117] "RemoveContainer" containerID="c98cbd4b215785997b751cbfcbea0367e4987945d0099617c8f031b71aa62268"
Jul 19 08:40:00.187749 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.187687    2559 generic.go:356] "Generic (PLEG): container finished" podID="38250c63-a420-4274-a5f2-29896f80892f" containerID="1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef" exitCode=6
Jul 19 08:40:00.188225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.187773    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" event={"ID":"38250c63-a420-4274-a5f2-29896f80892f","Type":"ContainerDied","Data":"1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef"}
Jul 19 08:40:00.188225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.187836    2559 scope.go:117] "RemoveContainer" containerID="c98cbd4b215785997b751cbfcbea0367e4987945d0099617c8f031b71aa62268"
Jul 19 08:40:00.188225 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.188027    2559 scope.go:117] "RemoveContainer" containerID="1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef"
Jul 19 08:40:00.188392 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:00.188254    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-b6flx_odh-ai-gateway-infra(38250c63-a420-4274-a5f2-29896f80892f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" podUID="38250c63-a420-4274-a5f2-29896f80892f"
Jul 19 08:40:00.734374 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.734340    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:00.734789 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.734775    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e567e4f-e72e-4a6d-b9c4-73a6b5644822" containerName="revoke-keys"
Jul 19 08:40:00.738898 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.738880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:00.743996 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.743971    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:00.831760 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.831696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc\" (UID: \"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:00.932722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.932676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc\" (UID: \"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:00.933279 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:00.933258    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc\" (UID: \"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:01.051483 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:01.051385    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:01.195535 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:01.195508    2559 scope.go:117] "RemoveContainer" containerID="1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef"
Jul 19 08:40:01.195924 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:01.195753    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-b6flx_odh-ai-gateway-infra(38250c63-a420-4274-a5f2-29896f80892f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" podUID="38250c63-a420-4274-a5f2-29896f80892f"
Jul 19 08:40:01.392411 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:01.392386    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:01.393281 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:40:01.393256    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd86211_9e4c_4fe4_8eb4_430b1a8eabc3.slice/crio-db02c5f561964fb897daf702b0e20e7615854990f9f53abf8f1096b92a3ec39b WatchSource:0}: Error finding container db02c5f561964fb897daf702b0e20e7615854990f9f53abf8f1096b92a3ec39b: Status 404 returned error can't find the container with id db02c5f561964fb897daf702b0e20e7615854990f9f53abf8f1096b92a3ec39b
Jul 19 08:40:02.202545 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:02.202508    2559 generic.go:356] "Generic (PLEG): container finished" podID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerID="244c0f0b3f6e5fade83963b4ba9bda44b7296c064172c458897bb932a515db37" exitCode=6
Jul 19 08:40:02.202995 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:02.202588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" event={"ID":"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3","Type":"ContainerDied","Data":"244c0f0b3f6e5fade83963b4ba9bda44b7296c064172c458897bb932a515db37"}
Jul 19 08:40:02.202995 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:02.202637    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" event={"ID":"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3","Type":"ContainerStarted","Data":"db02c5f561964fb897daf702b0e20e7615854990f9f53abf8f1096b92a3ec39b"}
Jul 19 08:40:02.202995 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:02.202831    2559 scope.go:117] "RemoveContainer" containerID="244c0f0b3f6e5fade83963b4ba9bda44b7296c064172c458897bb932a515db37"
Jul 19 08:40:03.209355 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:03.209325    2559 generic.go:356] "Generic (PLEG): container finished" podID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerID="d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9" exitCode=6
Jul 19 08:40:03.209807 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:03.209417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" event={"ID":"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3","Type":"ContainerDied","Data":"d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9"}
Jul 19 08:40:03.209807 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:03.209472    2559 scope.go:117] "RemoveContainer" containerID="244c0f0b3f6e5fade83963b4ba9bda44b7296c064172c458897bb932a515db37"
Jul 19 08:40:03.209807 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:03.209590    2559 scope.go:117] "RemoveContainer" containerID="d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9"
Jul 19 08:40:03.209940 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:03.209839    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc_odh-ai-gateway-infra(dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3"
Jul 19 08:40:04.217247 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:04.217212    2559 scope.go:117] "RemoveContainer" containerID="d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9"
Jul 19 08:40:04.217629 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:04.217430    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc_odh-ai-gateway-infra(dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3"
Jul 19 08:40:12.075470 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.075414    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"]
Jul 19 08:40:12.079957 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.079935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:12.085370 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.085341    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"]
Jul 19 08:40:12.238176 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.238136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr\" (UID: \"291a98a2-ede0-4705-80ae-9fcb3951fb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:12.338914 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.338823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr\" (UID: \"291a98a2-ede0-4705-80ae-9fcb3951fb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:12.339458 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.339439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr\" (UID: \"291a98a2-ede0-4705-80ae-9fcb3951fb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:12.391986 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.391954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:12.522923 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:12.522900    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"]
Jul 19 08:40:12.524851 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:40:12.524824    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291a98a2_ede0_4705_80ae_9fcb3951fb61.slice/crio-f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34 WatchSource:0}: Error finding container f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34: Status 404 returned error can't find the container with id f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34
Jul 19 08:40:13.267011 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.266973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr" event={"ID":"291a98a2-ede0-4705-80ae-9fcb3951fb61","Type":"ContainerStarted","Data":"55d60769c84e15df2a2e80cfe2022d51a27fc1b85598b5f2db42683b86af0dac"}
Jul 19 08:40:13.267400 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.267031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr" event={"ID":"291a98a2-ede0-4705-80ae-9fcb3951fb61","Type":"ContainerStarted","Data":"f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34"}
Jul 19 08:40:13.281223 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.281178    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr" podStartSLOduration=1.281163868 podStartE2EDuration="1.281163868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:40:13.279336 +0000 UTC m=+3449.788933894" watchObservedRunningTime="2026-07-19 08:40:13.281163868 +0000 UTC m=+3449.790761767"
Jul 19 08:40:13.988612 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.988576    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:13.991913 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.991897    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:13.998498 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:13.998451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:14.055820 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.055791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt\" (UID: \"af8425c2-69d9-4cf6-bb85-7d52a31fdce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:14.156936 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.156889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt\" (UID: \"af8425c2-69d9-4cf6-bb85-7d52a31fdce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:14.157495 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.157474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt\" (UID: \"af8425c2-69d9-4cf6-bb85-7d52a31fdce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:14.274463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.274366    2559 generic.go:356] "Generic (PLEG): container finished" podID="291a98a2-ede0-4705-80ae-9fcb3951fb61" containerID="55d60769c84e15df2a2e80cfe2022d51a27fc1b85598b5f2db42683b86af0dac" exitCode=0
Jul 19 08:40:14.274900 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.274450    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr" event={"ID":"291a98a2-ede0-4705-80ae-9fcb3951fb61","Type":"ContainerDied","Data":"55d60769c84e15df2a2e80cfe2022d51a27fc1b85598b5f2db42683b86af0dac"}
Jul 19 08:40:14.305354 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.305327    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:14.433239 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:14.433214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:14.434116 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:40:14.434089    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8425c2_69d9_4cf6_bb85_7d52a31fdce8.slice/crio-63911b9dee61bf3452781dae37ad533a94d8d8b5189732ab1f544bb8d4637162 WatchSource:0}: Error finding container 63911b9dee61bf3452781dae37ad533a94d8d8b5189732ab1f544bb8d4637162: Status 404 returned error can't find the container with id 63911b9dee61bf3452781dae37ad533a94d8d8b5189732ab1f544bb8d4637162
Jul 19 08:40:15.281645 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.281611    2559 generic.go:356] "Generic (PLEG): container finished" podID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerID="70489b2d0c75b7c6c6e5e856d51ea59b893599635447d7d877d0a464550f3d18" exitCode=6
Jul 19 08:40:15.282093 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.281712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" event={"ID":"af8425c2-69d9-4cf6-bb85-7d52a31fdce8","Type":"ContainerDied","Data":"70489b2d0c75b7c6c6e5e856d51ea59b893599635447d7d877d0a464550f3d18"}
Jul 19 08:40:15.282093 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.281795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" event={"ID":"af8425c2-69d9-4cf6-bb85-7d52a31fdce8","Type":"ContainerStarted","Data":"63911b9dee61bf3452781dae37ad533a94d8d8b5189732ab1f544bb8d4637162"}
Jul 19 08:40:15.282093 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.281898    2559 scope.go:117] "RemoveContainer" containerID="70489b2d0c75b7c6c6e5e856d51ea59b893599635447d7d877d0a464550f3d18"
Jul 19 08:40:15.427130 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.427106    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:15.469250 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.469215    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca\") pod \"291a98a2-ede0-4705-80ae-9fcb3951fb61\" (UID: \"291a98a2-ede0-4705-80ae-9fcb3951fb61\") "
Jul 19 08:40:15.469535 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.469513    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "291a98a2-ede0-4705-80ae-9fcb3951fb61" (UID: "291a98a2-ede0-4705-80ae-9fcb3951fb61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:40:15.570748 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:15.570689    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291a98a2-ede0-4705-80ae-9fcb3951fb61-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:40:16.288839 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.288753    2559 generic.go:356] "Generic (PLEG): container finished" podID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerID="7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792" exitCode=6
Jul 19 08:40:16.289246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.288834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" event={"ID":"af8425c2-69d9-4cf6-bb85-7d52a31fdce8","Type":"ContainerDied","Data":"7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792"}
Jul 19 08:40:16.289246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.288895    2559 scope.go:117] "RemoveContainer" containerID="70489b2d0c75b7c6c6e5e856d51ea59b893599635447d7d877d0a464550f3d18"
Jul 19 08:40:16.289246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.289058    2559 scope.go:117] "RemoveContainer" containerID="7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792"
Jul 19 08:40:16.289366 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:16.289273    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt_odh-ai-gateway-infra(af8425c2-69d9-4cf6-bb85-7d52a31fdce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8"
Jul 19 08:40:16.290404 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.290386    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr" event={"ID":"291a98a2-ede0-4705-80ae-9fcb3951fb61","Type":"ContainerDied","Data":"f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34"}
Jul 19 08:40:16.290495 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.290421    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e29bacd02483c20bac36309603c6afc0efd4077afe1f4ed2c102cf1fd4cb34"
Jul 19 08:40:16.290495 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.290423    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr"
Jul 19 08:40:16.461932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:16.461903    2559 scope.go:117] "RemoveContainer" containerID="1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef"
Jul 19 08:40:17.297442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:17.297353    2559 scope.go:117] "RemoveContainer" containerID="7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792"
Jul 19 08:40:17.297944 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:17.297591    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt_odh-ai-gateway-infra(af8425c2-69d9-4cf6-bb85-7d52a31fdce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8"
Jul 19 08:40:17.299015 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:17.298991    2559 generic.go:356] "Generic (PLEG): container finished" podID="38250c63-a420-4274-a5f2-29896f80892f" containerID="528e9633c45a5f1ffc5eea77ee9d2f00f65bceba00e6b7dc44854a11609dac38" exitCode=6
Jul 19 08:40:17.299146 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:17.299065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" event={"ID":"38250c63-a420-4274-a5f2-29896f80892f","Type":"ContainerDied","Data":"528e9633c45a5f1ffc5eea77ee9d2f00f65bceba00e6b7dc44854a11609dac38"}
Jul 19 08:40:17.299146 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:17.299129    2559 scope.go:117] "RemoveContainer" containerID="1820d68edbc4d028b009a34b50cdd000fefd4867ff16f07d1cafd415c3d3f4ef"
Jul 19 08:40:17.299354 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:17.299340    2559 scope.go:117] "RemoveContainer" containerID="528e9633c45a5f1ffc5eea77ee9d2f00f65bceba00e6b7dc44854a11609dac38"
Jul 19 08:40:17.299543 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:17.299521    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ef124b28-b6flx_odh-ai-gateway-infra(38250c63-a420-4274-a5f2-29896f80892f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" podUID="38250c63-a420-4274-a5f2-29896f80892f"
Jul 19 08:40:18.333057 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.333017    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:40:18.461488 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.461463    2559 scope.go:117] "RemoveContainer" containerID="d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9"
Jul 19 08:40:18.471188 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.471167    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:40:18.495626 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.495598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca\") pod \"38250c63-a420-4274-a5f2-29896f80892f\" (UID: \"38250c63-a420-4274-a5f2-29896f80892f\") "
Jul 19 08:40:18.495960 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.495937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38250c63-a420-4274-a5f2-29896f80892f" (UID: "38250c63-a420-4274-a5f2-29896f80892f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:40:18.597047 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:18.596960    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38250c63-a420-4274-a5f2-29896f80892f-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:40:19.313987 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.313956    2559 generic.go:356] "Generic (PLEG): container finished" podID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerID="2ad0f2f71b28b83b328ba918e670b76fd5eeb34b9da36ee9e5ee9eac50483da1" exitCode=6
Jul 19 08:40:19.314149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.314031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" event={"ID":"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3","Type":"ContainerDied","Data":"2ad0f2f71b28b83b328ba918e670b76fd5eeb34b9da36ee9e5ee9eac50483da1"}
Jul 19 08:40:19.314149 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.314101    2559 scope.go:117] "RemoveContainer" containerID="d89ce3f6e81833a8a07249b44248812c8a8aaf2e3c62808a921254241afcefe9"
Jul 19 08:40:19.314429 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.314412    2559 scope.go:117] "RemoveContainer" containerID="2ad0f2f71b28b83b328ba918e670b76fd5eeb34b9da36ee9e5ee9eac50483da1"
Jul 19 08:40:19.314736 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:19.314689    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc_odh-ai-gateway-infra(dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3"
Jul 19 08:40:19.315362 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.315339    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"
Jul 19 08:40:19.315362 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.315353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx" event={"ID":"38250c63-a420-4274-a5f2-29896f80892f","Type":"ContainerDied","Data":"bd09808cd32a6a60777e29ec34b34623ee416514f0468e8cf0a9c2326b289612"}
Jul 19 08:40:19.325514 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.325458    2559 scope.go:117] "RemoveContainer" containerID="528e9633c45a5f1ffc5eea77ee9d2f00f65bceba00e6b7dc44854a11609dac38"
Jul 19 08:40:19.344392 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.344369    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:40:19.350161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:19.350138    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef124b28-b6flx"]
Jul 19 08:40:20.339889 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.339860    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:20.460513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.460486    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38250c63-a420-4274-a5f2-29896f80892f" path="/var/lib/kubelet/pods/38250c63-a420-4274-a5f2-29896f80892f/volumes"
Jul 19 08:40:20.475068 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.475048    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:20.514573 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.514545    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca\") pod \"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3\" (UID: \"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3\") "
Jul 19 08:40:20.514907 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.514887    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" (UID: "dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:40:20.615953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:20.615871    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:40:21.329978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:21.329950    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"
Jul 19 08:40:21.329978 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:21.329973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc" event={"ID":"dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3","Type":"ContainerDied","Data":"db02c5f561964fb897daf702b0e20e7615854990f9f53abf8f1096b92a3ec39b"}
Jul 19 08:40:21.330167 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:21.330026    2559 scope.go:117] "RemoveContainer" containerID="2ad0f2f71b28b83b328ba918e670b76fd5eeb34b9da36ee9e5ee9eac50483da1"
Jul 19 08:40:21.351825 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:21.351801    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:21.355592 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:21.355570    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fdbad2e2-ttqsc"]
Jul 19 08:40:22.461156 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:22.461122    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" path="/var/lib/kubelet/pods/dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3/volumes"
Jul 19 08:40:31.455405 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:31.455369    2559 scope.go:117] "RemoveContainer" containerID="7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792"
Jul 19 08:40:32.395469 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:32.395385    2559 generic.go:356] "Generic (PLEG): container finished" podID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerID="c0ae3207616a02f707f04e5d633acd4a35a88ba4399a980f1be759075ccd0149" exitCode=6
Jul 19 08:40:32.395469 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:32.395422    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" event={"ID":"af8425c2-69d9-4cf6-bb85-7d52a31fdce8","Type":"ContainerDied","Data":"c0ae3207616a02f707f04e5d633acd4a35a88ba4399a980f1be759075ccd0149"}
Jul 19 08:40:32.395469 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:32.395470    2559 scope.go:117] "RemoveContainer" containerID="7bfb864e11de2457b94365167bde93dafcecdbdec5da90ea086975d2a1b0d792"
Jul 19 08:40:32.395865 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:32.395848    2559 scope.go:117] "RemoveContainer" containerID="c0ae3207616a02f707f04e5d633acd4a35a88ba4399a980f1be759075ccd0149"
Jul 19 08:40:32.396112 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:40:32.396092    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt_odh-ai-gateway-infra(af8425c2-69d9-4cf6-bb85-7d52a31fdce8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8"
Jul 19 08:40:33.439544 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:33.439507    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:33.571835 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:33.571806    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:33.631327 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:33.631288    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca\") pod \"af8425c2-69d9-4cf6-bb85-7d52a31fdce8\" (UID: \"af8425c2-69d9-4cf6-bb85-7d52a31fdce8\") "
Jul 19 08:40:33.631637 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:33.631616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "af8425c2-69d9-4cf6-bb85-7d52a31fdce8" (UID: "af8425c2-69d9-4cf6-bb85-7d52a31fdce8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:40:33.732246 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:33.732161    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af8425c2-69d9-4cf6-bb85-7d52a31fdce8-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:40:34.408737 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.408690    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"
Jul 19 08:40:34.408944 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.408692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt" event={"ID":"af8425c2-69d9-4cf6-bb85-7d52a31fdce8","Type":"ContainerDied","Data":"63911b9dee61bf3452781dae37ad533a94d8d8b5189732ab1f544bb8d4637162"}
Jul 19 08:40:34.408944 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.408843    2559 scope.go:117] "RemoveContainer" containerID="c0ae3207616a02f707f04e5d633acd4a35a88ba4399a980f1be759075ccd0149"
Jul 19 08:40:34.430627 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.430600    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:34.433721 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.433685    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2a60bf57-wgwxt"]
Jul 19 08:40:34.460975 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:34.460951    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" path="/var/lib/kubelet/pods/af8425c2-69d9-4cf6-bb85-7d52a31fdce8/volumes"
Jul 19 08:40:38.195173 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195134    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"]
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195527    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195556    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195561    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195575    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291a98a2-ede0-4705-80ae-9fcb3951fb61" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195580    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="291a98a2-ede0-4705-80ae-9fcb3951fb61" containerName="revoke-keys"
Jul 19 08:40:38.195586 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195588    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195594    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195608    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195614    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195621    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195626    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195631    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195635    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195642    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195647    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195716    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195725    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195732    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195737    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195743    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195751    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:40:38.195853 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.195759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="291a98a2-ede0-4705-80ae-9fcb3951fb61" containerName="revoke-keys"
Jul 19 08:40:38.200377 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.200359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:38.203311 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.203293    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:40:38.203446 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.203403    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:40:38.211573 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.211551    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"]
Jul 19 08:40:38.274073 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.274042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm\" (UID: \"d3f07a55-84b7-44df-8ad1-55ced956e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:38.374940 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.374906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm\" (UID: \"d3f07a55-84b7-44df-8ad1-55ced956e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:38.375576 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.375556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm\" (UID: \"d3f07a55-84b7-44df-8ad1-55ced956e8b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:38.511257 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.511173    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:38.640958 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:38.640934    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"]
Jul 19 08:40:38.642373 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:40:38.642337    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f07a55_84b7_44df_8ad1_55ced956e8b9.slice/crio-e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537 WatchSource:0}: Error finding container e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537: Status 404 returned error can't find the container with id e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537
Jul 19 08:40:39.437221 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:39.437182    2559 generic.go:356] "Generic (PLEG): container finished" podID="d3f07a55-84b7-44df-8ad1-55ced956e8b9" containerID="96e6bf73c0fa9814313f64a276330158e34e1592fc72646b0446c7ae5b40a090" exitCode=0
Jul 19 08:40:39.437566 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:39.437238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm" event={"ID":"d3f07a55-84b7-44df-8ad1-55ced956e8b9","Type":"ContainerDied","Data":"96e6bf73c0fa9814313f64a276330158e34e1592fc72646b0446c7ae5b40a090"}
Jul 19 08:40:39.437566 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:39.437316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm" event={"ID":"d3f07a55-84b7-44df-8ad1-55ced956e8b9","Type":"ContainerStarted","Data":"e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537"}
Jul 19 08:40:40.590867 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:40.590839    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:40.702064 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:40.701988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca\") pod \"d3f07a55-84b7-44df-8ad1-55ced956e8b9\" (UID: \"d3f07a55-84b7-44df-8ad1-55ced956e8b9\") "
Jul 19 08:40:40.702309 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:40.702286    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d3f07a55-84b7-44df-8ad1-55ced956e8b9" (UID: "d3f07a55-84b7-44df-8ad1-55ced956e8b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:40:40.802722 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:40.802674    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f07a55-84b7-44df-8ad1-55ced956e8b9-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:40:41.450607 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:41.450575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm" event={"ID":"d3f07a55-84b7-44df-8ad1-55ced956e8b9","Type":"ContainerDied","Data":"e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537"}
Jul 19 08:40:41.450797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:41.450612    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm"
Jul 19 08:40:41.450797 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:40:41.450621    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08a56e81a5ce0ef314b40c7ef18007e545c58fad9a029ceaec324d2777ce537"
Jul 19 08:41:10.588980 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:10.588903    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-mz9r9_a0057c50-2ac0-481a-96bd-f91c8a81cdee/copy-manifests/0.log"
Jul 19 08:41:10.595197 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:10.595174    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-mz9r9_a0057c50-2ac0-481a-96bd-f91c8a81cdee/manager/0.log"
Jul 19 08:41:11.159476 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:11.159445    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29740815-cl74l_7c92e219-7381-4ac8-bf38-b761d051b292/cleanup/0.log"
Jul 19 08:41:11.495135 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:11.495060    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-5da88c-t5d9n_c292ffa7-b861-461b-8fb8-ffabc2907004/revoke-keys/0.log"
Jul 19 08:41:11.821569 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:11.821539    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-49f9c5-wdvp2_c001200b-3ee8-4644-812d-a86a5d8db0dc/revoke-keys/0.log"
Jul 19 08:41:11.927220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:11.927190    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-916a44-fnhmg_ea10278a-a25b-4ed0-bdf0-a69d58de7838/revoke-keys/0.log"
Jul 19 08:41:12.034999 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:12.034970    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-07a46e45-x8xsw_48036c07-85c4-4084-bed3-dacec8924361/revoke-keys/0.log"
Jul 19 08:41:12.245767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:12.245681    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-72ce60-n9dhr_291a98a2-ede0-4705-80ae-9fcb3951fb61/revoke-keys/0.log"
Jul 19 08:41:12.350352 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:12.350325    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-190fde-8hxdm_d3f07a55-84b7-44df-8ad1-55ced956e8b9/revoke-keys/0.log"
Jul 19 08:41:13.175633 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.175606    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/util/0.log"
Jul 19 08:41:13.181103 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.181085    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/pull/0.log"
Jul 19 08:41:13.186513 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.186495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/extract/0.log"
Jul 19 08:41:13.292432 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.292404    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/extract/0.log"
Jul 19 08:41:13.298058 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.298037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/util/0.log"
Jul 19 08:41:13.303788 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.303771    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/pull/0.log"
Jul 19 08:41:13.414226 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.414198    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/util/0.log"
Jul 19 08:41:13.419992 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.419953    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/pull/0.log"
Jul 19 08:41:13.425748 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.425650    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/extract/0.log"
Jul 19 08:41:13.535767 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.535735    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/util/0.log"
Jul 19 08:41:13.541289 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.541266    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/pull/0.log"
Jul 19 08:41:13.547061 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:13.547043    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/extract/0.log"
Jul 19 08:41:14.771080 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:14.771047    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8rr4_97b5cadb-546b-4c39-9863-cbc4ef983810/istio-proxy/0.log"
Jul 19 08:41:14.882518 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:14.882490    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kph2_00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19/discovery/0.log"
Jul 19 08:41:15.226291 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.226248    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/1.log"
Jul 19 08:41:15.334879 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.334842    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-9h6dv_be1159f9-c573-4ef0-91b1-341a118ec325/payload-pre-processing/0.log"
Jul 19 08:41:15.729950 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.729917    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:15.730354 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730336    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:41:15.730442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730356    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:41:15.730442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730379    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f07a55-84b7-44df-8ad1-55ced956e8b9" containerName="revoke-keys"
Jul 19 08:41:15.730442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730387    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f07a55-84b7-44df-8ad1-55ced956e8b9" containerName="revoke-keys"
Jul 19 08:41:15.730442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730411    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:41:15.730442 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730420    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:41:15.730738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730526    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd86211-9e4c-4fe4-8eb4-430b1a8eabc3" containerName="revoke-keys"
Jul 19 08:41:15.730738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730539    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38250c63-a420-4274-a5f2-29896f80892f" containerName="revoke-keys"
Jul 19 08:41:15.730738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730550    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f07a55-84b7-44df-8ad1-55ced956e8b9" containerName="revoke-keys"
Jul 19 08:41:15.730738 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.730560    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af8425c2-69d9-4cf6-bb85-7d52a31fdce8" containerName="revoke-keys"
Jul 19 08:41:15.733939 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.733918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:15.737323 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.737303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 19 08:41:15.737596 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.737303    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbdd4\""
Jul 19 08:41:15.740223 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.740202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:15.800312 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.800274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g\" (UID: \"b88d53f5-cea3-49a3-848d-854e925e0e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:15.901457 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.901426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g\" (UID: \"b88d53f5-cea3-49a3-848d-854e925e0e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:15.902085 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.902066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g\" (UID: \"b88d53f5-cea3-49a3-848d-854e925e0e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:15.995439 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:15.995363    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wprbj_6964e5d9-9ca6-4632-aac1-3b01465f632e/main/0.log"
Jul 19 08:41:16.046561 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.046522    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:16.175574 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.175543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:16.178356 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:41:16.178330    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88d53f5_cea3_49a3_848d_854e925e0e07.slice/crio-d939be666cd42f8547b1e8bb87e386ce62ed9e9495c20c9f5bac4ae6c4e2ec0b WatchSource:0}: Error finding container d939be666cd42f8547b1e8bb87e386ce62ed9e9495c20c9f5bac4ae6c4e2ec0b: Status 404 returned error can't find the container with id d939be666cd42f8547b1e8bb87e386ce62ed9e9495c20c9f5bac4ae6c4e2ec0b
Jul 19 08:41:16.205056 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.205036    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db7jjtz_684b8471-aa95-4f16-b635-bdfdb84bf809/main/0.log"
Jul 19 08:41:16.310781 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.310756    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r4vmq_2fcaa8f2-5c86-435c-91b1-50b95cd4481c/main/0.log"
Jul 19 08:41:16.643243 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.643141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerStarted","Data":"64c57ecc77bcb5b1f24267c5c01b30262cd34bbbea6ca85c9ae64dd2319b79a1"}
Jul 19 08:41:16.643243 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.643197    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerStarted","Data":"d939be666cd42f8547b1e8bb87e386ce62ed9e9495c20c9f5bac4ae6c4e2ec0b"}
Jul 19 08:41:16.657932 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:16.657881    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" podStartSLOduration=1.657865551 podStartE2EDuration="1.657865551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 08:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:16.656657885 +0000 UTC m=+3513.166255783" watchObservedRunningTime="2026-07-19 08:41:16.657865551 +0000 UTC m=+3513.167463449"
Jul 19 08:41:17.650362 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:17.650332    2559 generic.go:356] "Generic (PLEG): container finished" podID="b88d53f5-cea3-49a3-848d-854e925e0e07" containerID="64c57ecc77bcb5b1f24267c5c01b30262cd34bbbea6ca85c9ae64dd2319b79a1" exitCode=6
Jul 19 08:41:17.650816 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:17.650421    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerDied","Data":"64c57ecc77bcb5b1f24267c5c01b30262cd34bbbea6ca85c9ae64dd2319b79a1"}
Jul 19 08:41:17.650816 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:17.650556    2559 scope.go:117] "RemoveContainer" containerID="64c57ecc77bcb5b1f24267c5c01b30262cd34bbbea6ca85c9ae64dd2319b79a1"
Jul 19 08:41:18.657659 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:18.657619    2559 generic.go:356] "Generic (PLEG): container finished" podID="b88d53f5-cea3-49a3-848d-854e925e0e07" containerID="b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2" exitCode=6
Jul 19 08:41:18.658157 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:18.657725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerDied","Data":"b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2"}
Jul 19 08:41:18.658157 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:18.657800    2559 scope.go:117] "RemoveContainer" containerID="64c57ecc77bcb5b1f24267c5c01b30262cd34bbbea6ca85c9ae64dd2319b79a1"
Jul 19 08:41:18.658157 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:18.658013    2559 scope.go:117] "RemoveContainer" containerID="b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2"
Jul 19 08:41:18.658312 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:41:18.658240    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g_odh-ai-gateway-infra(b88d53f5-cea3-49a3-848d-854e925e0e07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" podUID="b88d53f5-cea3-49a3-848d-854e925e0e07"
Jul 19 08:41:19.672103 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:19.672077    2559 scope.go:117] "RemoveContainer" containerID="b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2"
Jul 19 08:41:19.672505 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:41:19.672237    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g_odh-ai-gateway-infra(b88d53f5-cea3-49a3-848d-854e925e0e07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" podUID="b88d53f5-cea3-49a3-848d-854e925e0e07"
Jul 19 08:41:24.182501 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:24.182468    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p28rd_d90d5f08-c86a-42f2-a0d8-f9f316d0841f/global-pull-secret-syncer/0.log"
Jul 19 08:41:24.380083 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:24.380051    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vx5wp_1d332895-40aa-49e6-a1c2-33879f175472/konnectivity-agent/0.log"
Jul 19 08:41:24.397900 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:24.397870    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-91.ec2.internal_338760c60ab411de52b406a82ca30247/haproxy/0.log"
Jul 19 08:41:29.580139 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.580112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/extract/0.log"
Jul 19 08:41:29.605633 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.605604    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/util/0.log"
Jul 19 08:41:29.631591 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.631569    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7595hm9f_f316e89e-10f7-4a63-8b22-de9b8180f2d9/pull/0.log"
Jul 19 08:41:29.656421 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.656401    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/extract/0.log"
Jul 19 08:41:29.677507 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.677485    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/util/0.log"
Jul 19 08:41:29.697804 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.697785    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e08hw5p_1d8d8478-01f5-4341-9802-eb7138a2c7b3/pull/0.log"
Jul 19 08:41:29.721597 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.721571    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/extract/0.log"
Jul 19 08:41:29.739298 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.739269    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/util/0.log"
Jul 19 08:41:29.758097 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.758077    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7398klw_c20af166-298d-453d-8f22-f6f7048d8566/pull/0.log"
Jul 19 08:41:29.782239 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.782215    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/extract/0.log"
Jul 19 08:41:29.804126 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.804102    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/util/0.log"
Jul 19 08:41:29.820273 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:29.820235    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lz8dc_297dc5c4-78a4-4ff8-a940-0c99f99f847d/pull/0.log"
Jul 19 08:41:32.455794 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.455760    2559 scope.go:117] "RemoveContainer" containerID="b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2"
Jul 19 08:41:32.666128 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.666103    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzwdq_735e1b8b-d9a4-40b1-ad84-7e668521baf6/kube-state-metrics/0.log"
Jul 19 08:41:32.693820 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.693787    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzwdq_735e1b8b-d9a4-40b1-ad84-7e668521baf6/kube-rbac-proxy-main/0.log"
Jul 19 08:41:32.719742 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.719630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-nzwdq_735e1b8b-d9a4-40b1-ad84-7e668521baf6/kube-rbac-proxy-self/0.log"
Jul 19 08:41:32.750914 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.750881    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerStarted","Data":"3a0a0c27846f6084df2f820f1b7a34afaeeb8efddfdc9a9a1fb45eddef3a06c0"}
Jul 19 08:41:32.771551 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.771523    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-84bhj_cfc6dd51-1b56-428b-b17b-1e757727ec3a/monitoring-plugin/0.log"
Jul 19 08:41:32.966526 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.966495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfsv5_f348102a-107a-4ab9-b00e-5b69957c0a41/node-exporter/0.log"
Jul 19 08:41:32.987059 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:32.986999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfsv5_f348102a-107a-4ab9-b00e-5b69957c0a41/kube-rbac-proxy/0.log"
Jul 19 08:41:33.007055 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.007030    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pfsv5_f348102a-107a-4ab9-b00e-5b69957c0a41/init-textfile/0.log"
Jul 19 08:41:33.033231 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.033209    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nlh7n_86522d1a-24ec-499e-a1ed-6a36b9827c14/kube-rbac-proxy-main/0.log"
Jul 19 08:41:33.053178 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.053157    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nlh7n_86522d1a-24ec-499e-a1ed-6a36b9827c14/kube-rbac-proxy-self/0.log"
Jul 19 08:41:33.069980 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.069956    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-nlh7n_86522d1a-24ec-499e-a1ed-6a36b9827c14/openshift-state-metrics/0.log"
Jul 19 08:41:33.250888 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.250813    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-q9lxs_c0171d8f-66e5-42d9-84ae-fb9a09c33bf9/prometheus-operator/0.log"
Jul 19 08:41:33.267114 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.267086    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-q9lxs_c0171d8f-66e5-42d9-84ae-fb9a09c33bf9/kube-rbac-proxy/0.log"
Jul 19 08:41:33.288074 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.288055    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-kvc67_09a35b42-9b4b-40e9-bc01-47854539ab1b/prometheus-operator-admission-webhook/0.log"
Jul 19 08:41:33.315692 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.315671    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6c944bfd-n4dxs_000633b7-70d2-42c7-90ad-20c7235603a1/telemeter-client/0.log"
Jul 19 08:41:33.337953 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.337931    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6c944bfd-n4dxs_000633b7-70d2-42c7-90ad-20c7235603a1/reload/0.log"
Jul 19 08:41:33.358501 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.358482    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f6c944bfd-n4dxs_000633b7-70d2-42c7-90ad-20c7235603a1/kube-rbac-proxy/0.log"
Jul 19 08:41:33.757778 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.757746    2559 generic.go:356] "Generic (PLEG): container finished" podID="b88d53f5-cea3-49a3-848d-854e925e0e07" containerID="3a0a0c27846f6084df2f820f1b7a34afaeeb8efddfdc9a9a1fb45eddef3a06c0" exitCode=6
Jul 19 08:41:33.758198 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.757822    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerDied","Data":"3a0a0c27846f6084df2f820f1b7a34afaeeb8efddfdc9a9a1fb45eddef3a06c0"}
Jul 19 08:41:33.758198 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.757877    2559 scope.go:117] "RemoveContainer" containerID="b3f5d74fc0856780f37d1279112f6508cb62ddc07259c2d8427eddfa3ea28ea2"
Jul 19 08:41:33.758198 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.758177    2559 scope.go:117] "RemoveContainer" containerID="3a0a0c27846f6084df2f820f1b7a34afaeeb8efddfdc9a9a1fb45eddef3a06c0"
Jul 19 08:41:33.758383 ip-10-0-129-91 kubenswrapper[2559]: E0719 08:41:33.758362    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g_odh-ai-gateway-infra(b88d53f5-cea3-49a3-848d-854e925e0e07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" podUID="b88d53f5-cea3-49a3-848d-854e925e0e07"
Jul 19 08:41:33.781063 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:33.781038    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:34.447470 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.447435    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"]
Jul 19 08:41:34.452371 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.452351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.455364 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.455336    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"kube-root-ca.crt\""
Jul 19 08:41:34.456195 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.456172    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9nlzs\"/\"default-dockercfg-5jrnw\""
Jul 19 08:41:34.456279 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.456172    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9nlzs\"/\"openshift-service-ca.crt\""
Jul 19 08:41:34.462653 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.462632    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"]
Jul 19 08:41:34.557119 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.557087    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9v84\" (UniqueName: \"kubernetes.io/projected/e137fd3d-cad8-4f36-ba26-668d62fbc48d-kube-api-access-d9v84\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.557309 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.557277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-lib-modules\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.557395 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.557328    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-sys\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.557395 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.557374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-proc\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.557518 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.557399    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-podres\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658383 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9v84\" (UniqueName: \"kubernetes.io/projected/e137fd3d-cad8-4f36-ba26-668d62fbc48d-kube-api-access-d9v84\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-lib-modules\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-sys\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-proc\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-podres\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-sys\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658578 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-proc\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-podres\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.658817 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.658595    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e137fd3d-cad8-4f36-ba26-668d62fbc48d-lib-modules\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.665252 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.665226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9v84\" (UniqueName: \"kubernetes.io/projected/e137fd3d-cad8-4f36-ba26-668d62fbc48d-kube-api-access-d9v84\") pod \"perf-node-gather-daemonset-d2j74\" (UID: \"e137fd3d-cad8-4f36-ba26-668d62fbc48d\") " pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.763851 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.763768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:34.911570 ip-10-0-129-91 kubenswrapper[2559]: W0719 08:41:34.911542    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode137fd3d_cad8_4f36_ba26_668d62fbc48d.slice/crio-4c5c4e0f796736a3681d1c35cf8c21d51567bd6111f8a6e8d4f63da2a970fa10 WatchSource:0}: Error finding container 4c5c4e0f796736a3681d1c35cf8c21d51567bd6111f8a6e8d4f63da2a970fa10: Status 404 returned error can't find the container with id 4c5c4e0f796736a3681d1c35cf8c21d51567bd6111f8a6e8d4f63da2a970fa10
Jul 19 08:41:34.912166 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.912140    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"]
Jul 19 08:41:34.920196 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:34.920179    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:35.061009 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.060981    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca\") pod \"b88d53f5-cea3-49a3-848d-854e925e0e07\" (UID: \"b88d53f5-cea3-49a3-848d-854e925e0e07\") "
Jul 19 08:41:35.061340 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.061316    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b88d53f5-cea3-49a3-848d-854e925e0e07" (UID: "b88d53f5-cea3-49a3-848d-854e925e0e07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 19 08:41:35.162314 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.162279    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88d53f5-cea3-49a3-848d-854e925e0e07-maas-api-service-ca\") on node \"ip-10-0-129-91.ec2.internal\" DevicePath \"\""
Jul 19 08:41:35.707481 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.707450    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f5965b578-w7wxx_ec79ac7b-7cfc-4965-bcf6-b278837440ae/console/0.log"
Jul 19 08:41:35.741179 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.741153    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-cxgdv_88758f53-57e2-449e-8731-ee76a1f8efb9/download-server/0.log"
Jul 19 08:41:35.774838 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.774808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74" event={"ID":"e137fd3d-cad8-4f36-ba26-668d62fbc48d","Type":"ContainerStarted","Data":"fdc0f2f1eb6a8f45fdfde8c4d1a0c35cfce254c59b51be785ef2bb38c2269df9"}
Jul 19 08:41:35.775264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.774863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74" event={"ID":"e137fd3d-cad8-4f36-ba26-668d62fbc48d","Type":"ContainerStarted","Data":"4c5c4e0f796736a3681d1c35cf8c21d51567bd6111f8a6e8d4f63da2a970fa10"}
Jul 19 08:41:35.775264 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.774997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:35.776182 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.776160    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g" event={"ID":"b88d53f5-cea3-49a3-848d-854e925e0e07","Type":"ContainerDied","Data":"d939be666cd42f8547b1e8bb87e386ce62ed9e9495c20c9f5bac4ae6c4e2ec0b"}
Jul 19 08:41:35.776256 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.776198    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"
Jul 19 08:41:35.776256 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.776217    2559 scope.go:117] "RemoveContainer" containerID="3a0a0c27846f6084df2f820f1b7a34afaeeb8efddfdc9a9a1fb45eddef3a06c0"
Jul 19 08:41:35.792533 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.792494    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74" podStartSLOduration=1.7924812380000001 podStartE2EDuration="1.792481238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 08:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 08:41:35.791116486 +0000 UTC m=+3532.300714382" watchObservedRunningTime="2026-07-19 08:41:35.792481238 +0000 UTC m=+3532.302079167"
Jul 19 08:41:35.806848 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.806828    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:35.809053 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:35.809033    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1c28d394-wtq4g"]
Jul 19 08:41:36.460538 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:36.460505    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88d53f5-cea3-49a3-848d-854e925e0e07" path="/var/lib/kubelet/pods/b88d53f5-cea3-49a3-848d-854e925e0e07/volumes"
Jul 19 08:41:37.339000 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:37.338972    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5fdl_c686f16d-51d4-41e0-8096-e4c3ebf6d36a/dns/0.log"
Jul 19 08:41:37.358161 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:37.358136    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v5fdl_c686f16d-51d4-41e0-8096-e4c3ebf6d36a/kube-rbac-proxy/0.log"
Jul 19 08:41:37.405351 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:37.405318    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2dcs_0a0eb070-3b10-43df-950b-f9ac992e9049/dns-node-resolver/0.log"
Jul 19 08:41:38.009065 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:38.009037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7wmg_b47f2f7d-461c-40e1-9919-439c2dc8f02d/node-ca/0.log"
Jul 19 08:41:38.886555 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:38.886521    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8rr4_97b5cadb-546b-4c39-9863-cbc4ef983810/istio-proxy/0.log"
Jul 19 08:41:38.997644 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:38.997613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kph2_00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19/discovery/0.log"
Jul 19 08:41:39.068915 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:39.068886    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/0.log"
Jul 19 08:41:39.100010 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:39.099973    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/1.log"
Jul 19 08:41:39.128878 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:39.128852    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-9h6dv_be1159f9-c573-4ef0-91b1-341a118ec325/payload-pre-processing/0.log"
Jul 19 08:41:39.892332 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:39.892307    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rrlzv_15106dd6-44d1-474a-9c3c-d22f30a7d1b6/serve-healthcheck-canary/0.log"
Jul 19 08:41:40.517220 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:40.517195    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztphh_f708f7ac-d9ce-4aca-9dd4-0d144f8eb126/kube-rbac-proxy/0.log"
Jul 19 08:41:40.536920 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:40.536895    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztphh_f708f7ac-d9ce-4aca-9dd4-0d144f8eb126/exporter/0.log"
Jul 19 08:41:40.555407 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:40.555383    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ztphh_f708f7ac-d9ce-4aca-9dd4-0d144f8eb126/extractor/0.log"
Jul 19 08:41:41.794170 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:41.794143    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9nlzs/perf-node-gather-daemonset-d2j74"
Jul 19 08:41:43.375179 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:43.375150    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-mz9r9_a0057c50-2ac0-481a-96bd-f91c8a81cdee/manager/0.log"
Jul 19 08:41:43.396883 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:43.396858    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789467ff8c-mz9r9_a0057c50-2ac0-481a-96bd-f91c8a81cdee/copy-manifests/0.log"
Jul 19 08:41:44.763463 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:44.763381    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-747d9ff956-clwgr_8f1cb580-bb55-48cd-9be3-6120007a8630/manager/0.log"
Jul 19 08:41:44.808540 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:44.808513    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4knbx_7f9e950e-53df-475c-8e95-7aba1ee767eb/openshift-lws-operator/0.log"
Jul 19 08:41:45.395823 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:45.395801    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8rr4_97b5cadb-546b-4c39-9863-cbc4ef983810/istio-proxy/0.log"
Jul 19 08:41:45.510292 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:45.510253    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kph2_00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19/discovery/0.log"
Jul 19 08:41:45.557510 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:45.557480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/0.log"
Jul 19 08:41:45.588474 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:45.588450    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/1.log"
Jul 19 08:41:45.607356 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:45.607255    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-9h6dv_be1159f9-c573-4ef0-91b1-341a118ec325/payload-pre-processing/0.log"
Jul 19 08:41:46.840927 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:46.840899    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8rr4_97b5cadb-546b-4c39-9863-cbc4ef983810/istio-proxy/0.log"
Jul 19 08:41:46.951473 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:46.951433    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kph2_00fe4a5d-765b-4d27-a5d6-ef2ba62b0d19/discovery/0.log"
Jul 19 08:41:47.002274 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:47.002246    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/0.log"
Jul 19 08:41:47.030301 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:47.030274    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-9j6kr_628628a5-603d-4637-b044-99b658969e7c/istio-proxy/1.log"
Jul 19 08:41:47.050172 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:47.050150    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-9h6dv_be1159f9-c573-4ef0-91b1-341a118ec325/payload-pre-processing/0.log"
Jul 19 08:41:51.052969 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.052940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8z8rk_f392427d-b56a-4351-b5a2-47d8dac7c6a5/kube-multus/0.log"
Jul 19 08:41:51.076241 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.076217    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/kube-multus-additional-cni-plugins/0.log"
Jul 19 08:41:51.095022 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.094999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/egress-router-binary-copy/0.log"
Jul 19 08:41:51.112952 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.112926    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/cni-plugins/0.log"
Jul 19 08:41:51.132652 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.132630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/bond-cni-plugin/0.log"
Jul 19 08:41:51.152573 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.152546    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/routeoverride-cni/0.log"
Jul 19 08:41:51.170830 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.170806    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/whereabouts-cni-bincopy/0.log"
Jul 19 08:41:51.189671 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.189645    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jrn9b_5daba15c-5b35-477d-9154-ad2c0f2a108a/whereabouts-cni/0.log"
Jul 19 08:41:51.613726 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.613678    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xnkds_72fcf6a2-6fcc-4e80-b414-4d341f1e2719/network-metrics-daemon/0.log"
Jul 19 08:41:51.629585 ip-10-0-129-91 kubenswrapper[2559]: I0719 08:41:51.629561    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xnkds_72fcf6a2-6fcc-4e80-b414-4d341f1e2719/kube-rbac-proxy/0.log"